[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790284076800039-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 21:07:58 dist-test-slave-c7tk krb5kdc[399](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 21:07:58 dist-test-slave-c7tk krb5kdc[399](info): set up 2 sockets
Sep 24 21:07:58 dist-test-slave-c7tk krb5kdc[399](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260924 21:07:59.442458 392 mini_kdc.cc:121] Time spent starting KDC: real 2.625s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
W20260924 21:07:59.615470 392 mini_kdc.cc:253] Time spent creating user principal test-admin: real 0.173s user 0.001s sys 0.000s
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
W20260924 21:07:59.789644 392 mini_kdc.cc:253] Time spent creating user principal test-user: real 0.174s user 0.001s sys 0.000s
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
W20260924 21:07:59.936990 392 mini_kdc.cc:253] Time spent creating user principal joe-interloper: real 0.147s user 0.000s sys 0.002s
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790284076800039-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790284076800039-392-0/krb5kdc/test-user.keytab.
W20260924 21:08:00.569237 392 mini_kdc.cc:302] Time spent creating keytab for test-user: real 0.632s user 0.000s sys 0.001s
Sep 24 21:08:00 dist-test-slave-c7tk krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284080, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T21:08:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T21:08:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790284076800039-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790284076800039-392-0/krb5kdc/kudu_127.0.98.62.keytab.
W20260924 21:08:01.151798 392 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.0.98.62: real 0.562s user 0.000s sys 0.002s
I20260924 21:08:01.152246 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAzNNjS/build/release/bin/kudu
/tmp/dist-test-taskAzNNjS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790284076800039-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790284076800039-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790284076800039-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790284076800039-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39231
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790284076800039-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35597
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39231
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790284076800039-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790284076800039-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790284076800039-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 21:08:01.222674 412 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 21:08:01.222831 412 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 21:08:01.222849 412 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 21:08:01.223814 412 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 21:08:01.223850 412 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 21:08:01.223863 412 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 21:08:01.223874 412 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 21:08:01.223917 412 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 21:08:01.225248 412 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35597
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790284076800039-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790284076800039-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39231
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790284076800039-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39231
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790284076800039-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.412
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790284076800039-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 19:49:06 UTC on ccycloud.quasar-vhqjoc.root.comops.site
build id 4670864
I20260924 21:08:01.225485 412 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 21:08:01.225725 412 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 21:08:01.230521 417 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 21:08:01.230546 420 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 21:08:01.230582 418 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 21:08:01.230618 412 server_base.cc:1034] running on GCE node
I20260924 21:08:01.231000 412 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 21:08:01.231226 412 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 21:08:01.232411 412 hybrid_clock.cc:648] HybridClock initialized: now 1790284081232393 us; error 38 us; skew 500 ppm
Sep 24 21:08:01 dist-test-slave-c7tk krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284081, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 21:08:01.234864 412 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 21:08:01.235317 412 webserver.cc:460] Webserver started at http://127.0.98.62:33959/ using document root <none> and password file <none>
I20260924 21:08:01.235528 412 fs_manager.cc:359] Metadata directory not provided
I20260924 21:08:01.235592 412 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 21:08:01.235737 412 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 21:08:01.236716 412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790284076800039-392-0/minicluster-data/master-0/data/instance:
uuid: "8cfe582459684a0d8074801658c7ef7d"
format_stamp: "Formatted at 2026-09-24 21:08:01 on dist-test-slave-c7tk"
server_key: "dd62a891c9b8944b2d6dc2ddeae407d3"
server_key_iv: "bf164b7cba45c2bfb0ae84f5e226189d"
server_key_version: "encryptionkey@0"
I20260924 21:08:01.237068 412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790284076800039-392-0/minicluster-data/master-0/wal/instance:
uuid: "8cfe582459684a0d8074801658c7ef7d"
format_stamp: "Formatted at 2026-09-24 21:08:01 on dist-test-slave-c7tk"
server_key: "dd62a891c9b8944b2d6dc2ddeae407d3"
server_key_iv: "bf164b7cba45c2bfb0ae84f5e226189d"
server_key_version: "encryptionkey@0"
I20260924 21:08:01.238443 412 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260924 21:08:01.239133 427 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 21:08:01.239320 412 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 21:08:01.239428 412 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790284076800039-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790284076800039-392-0/minicluster-data/master-0/wal
uuid: "8cfe582459684a0d8074801658c7ef7d"
format_stamp: "Formatted at 2026-09-24 21:08:01 on dist-test-slave-c7tk"
server_key: "dd62a891c9b8944b2d6dc2ddeae407d3"
server_key_iv: "bf164b7cba45c2bfb0ae84f5e226189d"
server_key_version: "encryptionkey@0"
I20260924 21:08:01.239497 412 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790284076800039-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790284076800039-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790284076800039-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 21:08:01.266944 412 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 21:08:01.268853 412 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 21:08:01.269009 412 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 21:08:01.272356 412 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39231
I20260924 21:08:01.272382 479 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39231 every 8 connection(s)
I20260924 21:08:01.272686 412 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790284076800039-392-0/minicluster-data/master-0/data/info.pb
I20260924 21:08:01.273418 480 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 21:08:01.275295 480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8cfe582459684a0d8074801658c7ef7d: Bootstrap starting.
I20260924 21:08:01.275727 480 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8cfe582459684a0d8074801658c7ef7d: Neither blocks nor log segments found. Creating new log.
I20260924 21:08:01.275940 480 log.cc:824] T 00000000000000000000000000000000 P 8cfe582459684a0d8074801658c7ef7d: Log is configured to *not* fsync() on all Append() calls
I20260924 21:08:01.276470 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAzNNjS/build/release/bin/kudu as pid 412
I20260924 21:08:01.276576 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790284076800039-392-0/minicluster-data/master-0/wal/instance
I20260924 21:08:01.276698 480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8cfe582459684a0d8074801658c7ef7d: No bootstrap required, opened a new log
I20260924 21:08:01.277339 392 external_mini_cluster.cc:1428] Setting key f74882bbe392be610747e8f7c0ce2df9
I20260924 21:08:01.278788 480 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8cfe582459684a0d8074801658c7ef7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8cfe582459684a0d8074801658c7ef7d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39231 } }
I20260924 21:08:01.278932 480 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8cfe582459684a0d8074801658c7ef7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 21:08:01.278965 480 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8cfe582459684a0d8074801658c7ef7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8cfe582459684a0d8074801658c7ef7d, State: Initialized, Role: FOLLOWER
I20260924 21:08:01.279075 480 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8cfe582459684a0d8074801658c7ef7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8cfe582459684a0d8074801658c7ef7d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39231 } }
I20260924 21:08:01.279143 480 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8cfe582459684a0d8074801658c7ef7d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 21:08:01.279173 480 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8cfe582459684a0d8074801658c7ef7d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 21:08:01.279211 480 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8cfe582459684a0d8074801658c7ef7d [term 0 FOLLOWER]: Advancing to term 1
I20260924 21:08:01.280114 480 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8cfe582459684a0d8074801658c7ef7d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8cfe582459684a0d8074801658c7ef7d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39231 } }
I20260924 21:08:01.280221 480 leader_election.cc:302] T 00000000000000000000000000000000 P 8cfe582459684a0d8074801658c7ef7d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8cfe582459684a0d8074801658c7ef7d; no voters:
I20260924 21:08:01.280359 480 leader_election.cc:288] T 00000000000000000000000000000000 P 8cfe582459684a0d8074801658c7ef7d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 21:08:01.280380 485 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8cfe582459684a0d8074801658c7ef7d [term 1 FOLLOWER]: Leader election won for term 1
I20260924 21:08:01.280534 485 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8cfe582459684a0d8074801658c7ef7d [term 1 LEADER]: Becoming Leader. State: Replica: 8cfe582459684a0d8074801658c7ef7d, State: Running, Role: LEADER
I20260924 21:08:01.280577 480 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8cfe582459684a0d8074801658c7ef7d [sys.catalog]: configured and running, proceeding with master startup.
I20260924 21:08:01.280658 485 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8cfe582459684a0d8074801658c7ef7d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8cfe582459684a0d8074801658c7ef7d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39231 } }
I20260924 21:08:01.281036 486 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8cfe582459684a0d8074801658c7ef7d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8cfe582459684a0d8074801658c7ef7d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8cfe582459684a0d8074801658c7ef7d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39231 } } }
I20260924 21:08:01.281116 486 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8cfe582459684a0d8074801658c7ef7d [sys.catalog]: This master's current role is: LEADER
I20260924 21:08:01.281028 487 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8cfe582459684a0d8074801658c7ef7d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8cfe582459684a0d8074801658c7ef7d. Latest consensus state: current_term: 1 leader_uuid: "8cfe582459684a0d8074801658c7ef7d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8cfe582459684a0d8074801658c7ef7d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39231 } } }
I20260924 21:08:01.281236 487 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8cfe582459684a0d8074801658c7ef7d [sys.catalog]: This master's current role is: LEADER
I20260924 21:08:01.281368 494 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 21:08:01.281875 494 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 21:08:01.283331 494 catalog_manager.cc:1347] Generated new cluster ID: 2b33dd5400204271b0ccddfb2fe62043
I20260924 21:08:01.283383 494 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 24 21:08:01 dist-test-slave-c7tk krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284080, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 21:08:01.293367 484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:01.278160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33142 (local address 127.0.98.62:39231)
0924 21:08:01.280596 (+ 2436us) server_negotiation.cc:206] Beginning negotiation
0924 21:08:01.280603 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 21:08:01.280627 (+ 24us) server_negotiation.cc:407] Connection header received
0924 21:08:01.280716 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:08:01.280722 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:08:01.280774 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:08:01.280917 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:08:01.282915 (+ 1998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:01.284073 (+ 1158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:01.285361 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:01.285970 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:01.286075 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:01.289914 (+ 3839us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 21:08:01.289918 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 21:08:01.289919 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 21:08:01.289930 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 21:08:01.291634 (+ 1704us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:08:01.292040 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:08:01.292043 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:08:01.292043 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:08:01.292153 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:08:01.292386 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:08:01.292386 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:08:01.292387 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:08:01.292655 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 21:08:01.292790 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0924 21:08:01.293038 (+ 248us) server_negotiation.cc:299] Negotiation successful
0924 21:08:01.293158 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2400,"thread_start_us":76,"threads_started":1}
I20260924 21:08:01.300331 494 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 21:08:01.300783 494 catalog_manager.cc:1504] Loading token signing keys...
I20260924 21:08:01.304957 494 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8cfe582459684a0d8074801658c7ef7d: Generated new TSK 0
I20260924 21:08:01.305114 494 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790284076800039-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790284076800039-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260924 21:08:01.328469 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAzNNjS/build/release/bin/kudu
/tmp/dist-test-taskAzNNjS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790284076800039-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790284076800039-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790284076800039-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790284076800039-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39231
--keytab_file=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790284076800039-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35597
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790284076800039-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790284076800039-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790284076800039-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 21:08:01.402029 506 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 21:08:01.402207 506 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 21:08:01.402241 506 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 21:08:01.403230 506 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 21:08:01.403290 506 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 21:08:01.403330 506 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 21:08:01.404553 506 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35597
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790284076800039-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790284076800039-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790284076800039-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790284076800039-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39231
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.506
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790284076800039-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 19:49:06 UTC on ccycloud.quasar-vhqjoc.root.comops.site
build id 4670864
I20260924 21:08:01.404775 506 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 21:08:01.404969 506 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 21:08:01.409418 512 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 21:08:01.409466 511 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 21:08:01.409380 514 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 21:08:01.409556 506 server_base.cc:1034] running on GCE node
I20260924 21:08:01.409787 506 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 21:08:01.410030 506 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 21:08:01.411183 506 hybrid_clock.cc:648] HybridClock initialized: now 1790284081411174 us; error 24 us; skew 500 ppm
Sep 24 21:08:01 dist-test-slave-c7tk krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284081, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 21:08:01.413240 506 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 21:08:01.413654 506 webserver.cc:460] Webserver started at http://127.0.98.1:35793/ using document root <none> and password file <none>
I20260924 21:08:01.413921 506 fs_manager.cc:359] Metadata directory not provided
I20260924 21:08:01.413978 506 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 21:08:01.414099 506 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 21:08:01.415045 506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790284076800039-392-0/minicluster-data/ts-0/data/instance:
uuid: "6d89a61767974661a5d55472a74a097c"
format_stamp: "Formatted at 2026-09-24 21:08:01 on dist-test-slave-c7tk"
server_key: "fec7d15c0ef1e98108ec91626c5e9dd7"
server_key_iv: "67804d40c9db614048a56f16229a9f9e"
server_key_version: "encryptionkey@0"
I20260924 21:08:01.415355 506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790284076800039-392-0/minicluster-data/ts-0/wal/instance:
uuid: "6d89a61767974661a5d55472a74a097c"
format_stamp: "Formatted at 2026-09-24 21:08:01 on dist-test-slave-c7tk"
server_key: "fec7d15c0ef1e98108ec91626c5e9dd7"
server_key_iv: "67804d40c9db614048a56f16229a9f9e"
server_key_version: "encryptionkey@0"
I20260924 21:08:01.416536 506 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260924 21:08:01.417198 521 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 21:08:01.417379 506 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 21:08:01.417474 506 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790284076800039-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790284076800039-392-0/minicluster-data/ts-0/wal
uuid: "6d89a61767974661a5d55472a74a097c"
format_stamp: "Formatted at 2026-09-24 21:08:01 on dist-test-slave-c7tk"
server_key: "fec7d15c0ef1e98108ec91626c5e9dd7"
server_key_iv: "67804d40c9db614048a56f16229a9f9e"
server_key_version: "encryptionkey@0"
I20260924 21:08:01.417559 506 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790284076800039-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790284076800039-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790284076800039-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 21:08:01.443677 506 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 21:08:01.445482 506 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 21:08:01.445618 506 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 21:08:01.445832 506 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 21:08:01.446090 506 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 21:08:01.446137 506 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 21:08:01.446168 506 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 21:08:01.446192 506 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 21:08:01.452075 506 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:43903
I20260924 21:08:01.452119 634 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:43903 every 8 connection(s)
I20260924 21:08:01.452378 506 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790284076800039-392-0/minicluster-data/ts-0/data/info.pb
Sep 24 21:08:01 dist-test-slave-c7tk krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284081, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 21:08:01.459468 484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:01.452968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:50191 (local address 127.0.98.62:39231)
0924 21:08:01.453022 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0924 21:08:01.453024 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 21:08:01.453141 (+ 117us) server_negotiation.cc:407] Connection header received
0924 21:08:01.453493 (+ 352us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:08:01.453493 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:08:01.453501 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:08:01.453543 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:08:01.454309 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:01.454722 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:01.455498 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:01.455646 (+ 148us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:01.455688 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:01.457498 (+ 1810us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 21:08:01.457499 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 21:08:01.457499 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 21:08:01.457504 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 21:08:01.458549 (+ 1045us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:08:01.458839 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:08:01.458840 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:08:01.458840 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:08:01.458883 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:08:01.459061 (+ 178us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:08:01.459061 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:08:01.459061 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:08:01.459140 (+ 79us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 21:08:01.459186 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0924 21:08:01.459359 (+ 173us) server_negotiation.cc:299] Negotiation successful
0924 21:08:01.459415 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260924 21:08:01.459642 637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:01.452915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39231 (local address 127.0.98.1:50191)
0924 21:08:01.453046 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0924 21:08:01.453077 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0924 21:08:01.453428 (+ 351us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 21:08:01.453589 (+ 161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 21:08:01.453592 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 21:08:01.454099 (+ 507us) client_negotiation.cc:189] Negotiated authn=SASL
0924 21:08:01.454245 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:01.454249 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:01.454786 (+ 537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:01.454786 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:01.455434 (+ 648us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:01.455435 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:01.455704 (+ 269us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:01.455704 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:01.455780 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:01.456183 (+ 403us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 21:08:01.456199 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 21:08:01.457417 (+ 1218us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 21:08:01.458614 (+ 1197us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 21:08:01.458619 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 21:08:01.458620 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 21:08:01.458785 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 21:08:01.458941 (+ 156us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 21:08:01.458941 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 21:08:01.458941 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 21:08:01.459015 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 21:08:01.459186 (+ 171us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 21:08:01.459188 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 21:08:01.459273 (+ 85us) client_negotiation.cc:769] Sending connection context
0924 21:08:01.459336 (+ 63us) client_negotiation.cc:240] Negotiation successful
0924 21:08:01.459426 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":85,"thread_start_us":63,"threads_started":1}
I20260924 21:08:01.460091 635 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39231
I20260924 21:08:01.460176 635 heartbeater.cc:464] Registering TS with master...
I20260924 21:08:01.460325 635 heartbeater.cc:511] Master 127.0.98.62:39231 requested a full tablet report, sending...
I20260924 21:08:01.460733 444 ts_manager.cc:194] Registered new tserver with Master: 6d89a61767974661a5d55472a74a097c (127.0.98.1:43903)
I20260924 21:08:01.461181 444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:50191
I20260924 21:08:01.462369 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAzNNjS/build/release/bin/kudu as pid 506
I20260924 21:08:01.462436 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790284076800039-392-0/minicluster-data/ts-0/wal/instance
I20260924 21:08:01.462523 392 external_mini_cluster.cc:1428] Setting key d4edfb7624dbc3ab22c6bb484674b7fd
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790284076800039-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790284076800039-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260924 21:08:01.483736 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAzNNjS/build/release/bin/kudu
/tmp/dist-test-taskAzNNjS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790284076800039-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790284076800039-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790284076800039-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790284076800039-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39231
--keytab_file=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790284076800039-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35597
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790284076800039-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790284076800039-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790284076800039-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 21:08:01.550697 640 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 21:08:01.550865 640 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 21:08:01.550885 640 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 21:08:01.551873 640 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 21:08:01.551909 640 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 21:08:01.552222 640 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 21:08:01.553650 640 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35597
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790284076800039-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790284076800039-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790284076800039-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790284076800039-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39231
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.640
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790284076800039-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 19:49:06 UTC on ccycloud.quasar-vhqjoc.root.comops.site
build id 4670864
I20260924 21:08:01.553906 640 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 21:08:01.554117 640 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 21:08:01.558519 646 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 21:08:01.558626 645 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 21:08:01.558671 640 server_base.cc:1034] running on GCE node
W20260924 21:08:01.558550 648 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 21:08:01.558884 640 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 21:08:01.559095 640 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 21:08:01.560217 640 hybrid_clock.cc:648] HybridClock initialized: now 1790284081560209 us; error 24 us; skew 500 ppm
Sep 24 21:08:01 dist-test-slave-c7tk krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284081, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 21:08:01.562398 640 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 21:08:01.562773 640 webserver.cc:460] Webserver started at http://127.0.98.2:43577/ using document root <none> and password file <none>
I20260924 21:08:01.562951 640 fs_manager.cc:359] Metadata directory not provided
I20260924 21:08:01.562988 640 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 21:08:01.563114 640 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 21:08:01.563961 640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790284076800039-392-0/minicluster-data/ts-1/data/instance:
uuid: "a83306ddbbe241d0923f9097a6850322"
format_stamp: "Formatted at 2026-09-24 21:08:01 on dist-test-slave-c7tk"
server_key: "b5dd2724a2108f706c7e989ba378c925"
server_key_iv: "edb0be0cf9dd636408eb84a62e15a293"
server_key_version: "encryptionkey@0"
I20260924 21:08:01.564266 640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790284076800039-392-0/minicluster-data/ts-1/wal/instance:
uuid: "a83306ddbbe241d0923f9097a6850322"
format_stamp: "Formatted at 2026-09-24 21:08:01 on dist-test-slave-c7tk"
server_key: "b5dd2724a2108f706c7e989ba378c925"
server_key_iv: "edb0be0cf9dd636408eb84a62e15a293"
server_key_version: "encryptionkey@0"
I20260924 21:08:01.565466 640 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 21:08:01.566179 655 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 21:08:01.566381 640 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 21:08:01.566465 640 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790284076800039-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790284076800039-392-0/minicluster-data/ts-1/wal
uuid: "a83306ddbbe241d0923f9097a6850322"
format_stamp: "Formatted at 2026-09-24 21:08:01 on dist-test-slave-c7tk"
server_key: "b5dd2724a2108f706c7e989ba378c925"
server_key_iv: "edb0be0cf9dd636408eb84a62e15a293"
server_key_version: "encryptionkey@0"
I20260924 21:08:01.566532 640 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790284076800039-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790284076800039-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790284076800039-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 21:08:01.585424 640 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 21:08:01.587412 640 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 21:08:01.587568 640 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 21:08:01.587762 640 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 21:08:01.588006 640 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 21:08:01.588033 640 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 21:08:01.588052 640 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 21:08:01.588080 640 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 21:08:01.594116 640 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:46513
I20260924 21:08:01.594177 768 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:46513 every 8 connection(s)
I20260924 21:08:01.594429 640 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790284076800039-392-0/minicluster-data/ts-1/data/info.pb
I20260924 21:08:01.597133 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAzNNjS/build/release/bin/kudu as pid 640
I20260924 21:08:01.597223 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790284076800039-392-0/minicluster-data/ts-1/wal/instance
I20260924 21:08:01.597373 392 external_mini_cluster.cc:1428] Setting key 9ff70d0e883aa55a4654b2b18952e30f
Sep 24 21:08:01 dist-test-slave-c7tk krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284081, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 21:08:01.602743 484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:01.594974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:52405 (local address 127.0.98.62:39231)
0924 21:08:01.595029 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0924 21:08:01.595031 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 21:08:01.595204 (+ 173us) server_negotiation.cc:407] Connection header received
0924 21:08:01.595555 (+ 351us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:08:01.595556 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:08:01.595564 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:08:01.595613 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:08:01.596360 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:01.596832 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:01.598219 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:01.598381 (+ 162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:01.598425 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:01.600275 (+ 1850us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 21:08:01.600277 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 21:08:01.600277 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 21:08:01.600284 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 21:08:01.601484 (+ 1200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:08:01.601920 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:08:01.601921 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:08:01.601921 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:08:01.601970 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:08:01.602220 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:08:01.602220 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:08:01.602220 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:08:01.602302 (+ 82us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 21:08:01.602345 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0924 21:08:01.602636 (+ 291us) server_negotiation.cc:299] Negotiation successful
0924 21:08:01.602692 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260924 21:08:01.602981 771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:01.594961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39231 (local address 127.0.98.2:52405)
0924 21:08:01.595098 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0924 21:08:01.595129 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0924 21:08:01.595485 (+ 356us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 21:08:01.595660 (+ 175us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 21:08:01.595663 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 21:08:01.596101 (+ 438us) client_negotiation.cc:189] Negotiated authn=SASL
0924 21:08:01.596264 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:01.596267 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:01.596923 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:01.596924 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:01.598142 (+ 1218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:01.598144 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:01.598449 (+ 305us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:01.598450 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:01.598569 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:01.598948 (+ 379us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 21:08:01.598956 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 21:08:01.600194 (+ 1238us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 21:08:01.601566 (+ 1372us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 21:08:01.601572 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 21:08:01.601573 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 21:08:01.601844 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 21:08:01.602032 (+ 188us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 21:08:01.602033 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 21:08:01.602034 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 21:08:01.602157 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 21:08:01.602358 (+ 201us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 21:08:01.602363 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 21:08:01.602514 (+ 151us) client_negotiation.cc:769] Sending connection context
0924 21:08:01.602622 (+ 108us) client_negotiation.cc:240] Negotiation successful
0924 21:08:01.602771 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":85,"threads_started":1}
I20260924 21:08:01.603358 769 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39231
I20260924 21:08:01.603441 769 heartbeater.cc:464] Registering TS with master...
I20260924 21:08:01.603585 769 heartbeater.cc:511] Master 127.0.98.62:39231 requested a full tablet report, sending...
I20260924 21:08:01.603861 444 ts_manager.cc:194] Registered new tserver with Master: a83306ddbbe241d0923f9097a6850322 (127.0.98.2:46513)
I20260924 21:08:01.604174 444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:52405
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790284076800039-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790284076800039-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260924 21:08:01.619890 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAzNNjS/build/release/bin/kudu
/tmp/dist-test-taskAzNNjS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790284076800039-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790284076800039-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790284076800039-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790284076800039-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39231
--keytab_file=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790284076800039-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35597
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790284076800039-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790284076800039-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790284076800039-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 21:08:01.690282 774 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 21:08:01.690438 774 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 21:08:01.690456 774 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 21:08:01.691398 774 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 21:08:01.691435 774 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 21:08:01.691457 774 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 21:08:01.692629 774 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35597
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790284076800039-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790284076800039-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790284076800039-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790284076800039-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39231
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.774
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790284076800039-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 19:49:06 UTC on ccycloud.quasar-vhqjoc.root.comops.site
build id 4670864
I20260924 21:08:01.692829 774 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 21:08:01.693017 774 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 21:08:01.697260 782 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 21:08:01.697374 774 server_base.cc:1034] running on GCE node
W20260924 21:08:01.697314 780 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 21:08:01.697504 779 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 21:08:01.697671 774 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 21:08:01.697927 774 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 21:08:01.699077 774 hybrid_clock.cc:648] HybridClock initialized: now 1790284081699060 us; error 32 us; skew 500 ppm
Sep 24 21:08:01 dist-test-slave-c7tk krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284081, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 21:08:01.701151 774 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 21:08:01.701511 774 webserver.cc:460] Webserver started at http://127.0.98.3:35409/ using document root <none> and password file <none>
I20260924 21:08:01.701668 774 fs_manager.cc:359] Metadata directory not provided
I20260924 21:08:01.701701 774 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 21:08:01.701772 774 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 21:08:01.702587 774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790284076800039-392-0/minicluster-data/ts-2/data/instance:
uuid: "909fd34659ec4d8fbc5b54cd87e95808"
format_stamp: "Formatted at 2026-09-24 21:08:01 on dist-test-slave-c7tk"
server_key: "fbbcf8f4c2baab5e14881a93d5edac38"
server_key_iv: "0521ef67f5f2906166c49f243879ec53"
server_key_version: "encryptionkey@0"
I20260924 21:08:01.702961 774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790284076800039-392-0/minicluster-data/ts-2/wal/instance:
uuid: "909fd34659ec4d8fbc5b54cd87e95808"
format_stamp: "Formatted at 2026-09-24 21:08:01 on dist-test-slave-c7tk"
server_key: "fbbcf8f4c2baab5e14881a93d5edac38"
server_key_iv: "0521ef67f5f2906166c49f243879ec53"
server_key_version: "encryptionkey@0"
I20260924 21:08:01.704164 774 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20260924 21:08:01.704708 789 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 21:08:01.704907 774 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 21:08:01.704968 774 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790284076800039-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790284076800039-392-0/minicluster-data/ts-2/wal
uuid: "909fd34659ec4d8fbc5b54cd87e95808"
format_stamp: "Formatted at 2026-09-24 21:08:01 on dist-test-slave-c7tk"
server_key: "fbbcf8f4c2baab5e14881a93d5edac38"
server_key_iv: "0521ef67f5f2906166c49f243879ec53"
server_key_version: "encryptionkey@0"
I20260924 21:08:01.705008 774 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790284076800039-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790284076800039-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790284076800039-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 21:08:01.720038 774 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 21:08:01.721410 774 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 21:08:01.721495 774 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 21:08:01.721617 774 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 21:08:01.721813 774 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 21:08:01.721838 774 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 21:08:01.721855 774 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 21:08:01.721870 774 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 21:08:01.727373 774 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:45135
I20260924 21:08:01.727440 902 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:45135 every 8 connection(s)
I20260924 21:08:01.727681 774 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790284076800039-392-0/minicluster-data/ts-2/data/info.pb
Sep 24 21:08:01 dist-test-slave-c7tk krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284081, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 21:08:01.734059 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAzNNjS/build/release/bin/kudu as pid 774
I20260924 21:08:01.734153 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790284076800039-392-0/minicluster-data/ts-2/wal/instance
I20260924 21:08:01.734297 392 external_mini_cluster.cc:1428] Setting key d196d2dee89081743ea230b9ffc78612
I20260924 21:08:01.735504 484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:01.728265 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:58437 (local address 127.0.98.62:39231)
0924 21:08:01.728316 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0924 21:08:01.728318 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 21:08:01.728456 (+ 138us) server_negotiation.cc:407] Connection header received
0924 21:08:01.728869 (+ 413us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:08:01.728870 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:08:01.728890 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:08:01.728938 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:08:01.729722 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:01.730223 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:01.731043 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:01.731233 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:01.731312 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:01.733069 (+ 1757us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 21:08:01.733071 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 21:08:01.733071 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 21:08:01.733078 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 21:08:01.734221 (+ 1143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:08:01.734621 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:08:01.734622 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:08:01.734622 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:08:01.734679 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:08:01.734927 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:08:01.734928 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:08:01.734928 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:08:01.735049 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 21:08:01.735130 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0924 21:08:01.735391 (+ 261us) server_negotiation.cc:299] Negotiation successful
0924 21:08:01.735452 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260924 21:08:01.735589 905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:01.728234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39231 (local address 127.0.98.3:58437)
0924 21:08:01.728360 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0924 21:08:01.728394 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0924 21:08:01.728745 (+ 351us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 21:08:01.728998 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 21:08:01.729001 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 21:08:01.729471 (+ 470us) client_negotiation.cc:189] Negotiated authn=SASL
0924 21:08:01.729627 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:01.729630 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:01.730303 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:01.730303 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:01.730956 (+ 653us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:01.730958 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:01.731320 (+ 362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:01.731320 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:01.731432 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:01.731782 (+ 350us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 21:08:01.731788 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0924 21:08:01.732981 (+ 1193us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 21:08:01.734288 (+ 1307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 21:08:01.734293 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 21:08:01.734295 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 21:08:01.734555 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 21:08:01.734748 (+ 193us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 21:08:01.734749 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 21:08:01.734750 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 21:08:01.734864 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 21:08:01.735133 (+ 269us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 21:08:01.735137 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 21:08:01.735264 (+ 127us) client_negotiation.cc:769] Sending connection context
0924 21:08:01.735351 (+ 87us) client_negotiation.cc:240] Negotiation successful
0924 21:08:01.735453 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":92,"thread_start_us":64,"threads_started":1}
I20260924 21:08:01.735953 903 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39231
I20260924 21:08:01.736012 903 heartbeater.cc:464] Registering TS with master...
I20260924 21:08:01.736109 903 heartbeater.cc:511] Master 127.0.98.62:39231 requested a full tablet report, sending...
I20260924 21:08:01.736382 444 ts_manager.cc:194] Registered new tserver with Master: 909fd34659ec4d8fbc5b54cd87e95808 (127.0.98.3:45135)
I20260924 21:08:01.736739 444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:58437
I20260924 21:08:01.746004 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 24 21:08:01 dist-test-slave-c7tk krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284081, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 24 21:08:01 dist-test-slave-c7tk krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284081, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260924 21:08:01.765329 910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:01.758655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40608 (local address 127.0.98.1:43903)
0924 21:08:01.759238 (+ 583us) server_negotiation.cc:206] Beginning negotiation
0924 21:08:01.759241 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 21:08:01.759250 (+ 9us) server_negotiation.cc:407] Connection header received
0924 21:08:01.759270 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:08:01.759271 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:08:01.759293 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:08:01.759347 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:08:01.759796 (+ 449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:01.760431 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:01.761034 (+ 603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:01.761285 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:01.761340 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:01.763219 (+ 1879us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 21:08:01.763225 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 21:08:01.763226 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 21:08:01.763232 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 21:08:01.764428 (+ 1196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:08:01.764704 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:08:01.764706 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:08:01.764706 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:08:01.764756 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:08:01.764909 (+ 153us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:08:01.764909 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:08:01.764909 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:08:01.765005 (+ 96us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 21:08:01.765094 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0924 21:08:01.765146 (+ 52us) server_negotiation.cc:299] Negotiation successful
0924 21:08:01.765201 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":552,"thread_start_us":506,"threads_started":1}
W20260924 21:08:01.765656 549 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40608
Sep 24 21:08:01 dist-test-slave-c7tk krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284081, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 24 21:08:01 dist-test-slave-c7tk krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284081, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260924 21:08:01.783483 910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:01.777903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40618 (local address 127.0.98.1:43903)
0924 21:08:01.777972 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0924 21:08:01.777973 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 21:08:01.778054 (+ 81us) server_negotiation.cc:407] Connection header received
0924 21:08:01.778085 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:08:01.778085 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:08:01.778093 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:08:01.778138 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:08:01.778573 (+ 435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:01.778977 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:01.779532 (+ 555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:01.779698 (+ 166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:01.779742 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:01.781560 (+ 1818us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 21:08:01.781561 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 21:08:01.781561 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 21:08:01.781566 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 21:08:01.782729 (+ 1163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:08:01.782986 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:08:01.782986 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:08:01.782987 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:08:01.783035 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:08:01.783172 (+ 137us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:08:01.783172 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:08:01.783172 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:08:01.783241 (+ 69us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 21:08:01.783314 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0924 21:08:01.783377 (+ 63us) server_negotiation.cc:299] Negotiation successful
0924 21:08:01.783433 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260924 21:08:01.783895 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAzNNjS/build/release/bin/kudu with pid 506
I20260924 21:08:01.788278 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAzNNjS/build/release/bin/kudu with pid 640
I20260924 21:08:01.793525 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAzNNjS/build/release/bin/kudu with pid 774
I20260924 21:08:01.797294 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAzNNjS/build/release/bin/kudu with pid 412
2026-09-24T21:08:01Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (5004 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790284076800039-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 21:08:01 dist-test-slave-c7tk krb5kdc[917](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 21:08:01 dist-test-slave-c7tk krb5kdc[917](info): set up 2 sockets
Sep 24 21:08:01 dist-test-slave-c7tk krb5kdc[917](info): commencing operation
krb5kdc: starting...
W20260924 21:08:02.724769 392 mini_kdc.cc:121] Time spent starting KDC: real 0.908s user 0.000s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790284076800039-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790284076800039-392-0/krb5kdc/test-user.keytab.
Sep 24 21:08:02 dist-test-slave-c7tk krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284082, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T21:08:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T21:08:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790284076800039-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790284076800039-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 21:08:02.824247 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAzNNjS/build/release/bin/kudu
/tmp/dist-test-taskAzNNjS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790284076800039-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790284076800039-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790284076800039-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790284076800039-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35133
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790284076800039-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39391
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35133
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790284076800039-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790284076800039-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790284076800039-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 21:08:02.893054 930 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 21:08:02.893254 930 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 21:08:02.893285 930 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 21:08:02.894327 930 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 21:08:02.894394 930 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 21:08:02.894418 930 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 21:08:02.894444 930 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 21:08:02.894464 930 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 21:08:02.895718 930 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39391
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790284076800039-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790284076800039-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35133
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790284076800039-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35133
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790284076800039-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.930
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790284076800039-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 19:49:06 UTC on ccycloud.quasar-vhqjoc.root.comops.site
build id 4670864
I20260924 21:08:02.895957 930 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 21:08:02.896153 930 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 21:08:02.900529 938 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 21:08:02.900676 930 server_base.cc:1034] running on GCE node
W20260924 21:08:02.900657 936 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 21:08:02.900657 935 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 21:08:02.901029 930 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 21:08:02.901281 930 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 21:08:02.902419 930 hybrid_clock.cc:648] HybridClock initialized: now 1790284082902408 us; error 40 us; skew 500 ppm
Sep 24 21:08:02 dist-test-slave-c7tk krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284082, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 21:08:02.904428 930 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 21:08:02.904806 930 webserver.cc:460] Webserver started at http://127.0.98.62:34585/ using document root <none> and password file <none>
I20260924 21:08:02.905006 930 fs_manager.cc:359] Metadata directory not provided
I20260924 21:08:02.905066 930 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 21:08:02.905175 930 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 21:08:02.906108 930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790284076800039-392-0/minicluster-data/master-0/data/instance:
uuid: "2fedf5e0451b42bebbef2c882bcf2cd6"
format_stamp: "Formatted at 2026-09-24 21:08:02 on dist-test-slave-c7tk"
server_key: "4f52c437ea56bca39478ec59bfb6f225"
server_key_iv: "09c4d23b17979a9b5acd8cbf31fa6dd5"
server_key_version: "encryptionkey@0"
I20260924 21:08:02.906430 930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790284076800039-392-0/minicluster-data/master-0/wal/instance:
uuid: "2fedf5e0451b42bebbef2c882bcf2cd6"
format_stamp: "Formatted at 2026-09-24 21:08:02 on dist-test-slave-c7tk"
server_key: "4f52c437ea56bca39478ec59bfb6f225"
server_key_iv: "09c4d23b17979a9b5acd8cbf31fa6dd5"
server_key_version: "encryptionkey@0"
I20260924 21:08:02.907658 930 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 21:08:02.908277 945 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 21:08:02.908430 930 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 21:08:02.908519 930 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790284076800039-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790284076800039-392-0/minicluster-data/master-0/wal
uuid: "2fedf5e0451b42bebbef2c882bcf2cd6"
format_stamp: "Formatted at 2026-09-24 21:08:02 on dist-test-slave-c7tk"
server_key: "4f52c437ea56bca39478ec59bfb6f225"
server_key_iv: "09c4d23b17979a9b5acd8cbf31fa6dd5"
server_key_version: "encryptionkey@0"
I20260924 21:08:02.908586 930 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790284076800039-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790284076800039-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790284076800039-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 21:08:02.947402 930 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 21:08:02.949019 930 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 21:08:02.949177 930 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 21:08:02.953013 930 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35133
I20260924 21:08:02.953073 997 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35133 every 8 connection(s)
I20260924 21:08:02.953400 930 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790284076800039-392-0/minicluster-data/master-0/data/info.pb
I20260924 21:08:02.954288 998 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 21:08:02.956252 998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2fedf5e0451b42bebbef2c882bcf2cd6: Bootstrap starting.
I20260924 21:08:02.956812 998 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2fedf5e0451b42bebbef2c882bcf2cd6: Neither blocks nor log segments found. Creating new log.
I20260924 21:08:02.957099 998 log.cc:824] T 00000000000000000000000000000000 P 2fedf5e0451b42bebbef2c882bcf2cd6: Log is configured to *not* fsync() on all Append() calls
I20260924 21:08:02.957743 998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2fedf5e0451b42bebbef2c882bcf2cd6: No bootstrap required, opened a new log
I20260924 21:08:02.957967 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAzNNjS/build/release/bin/kudu as pid 930
I20260924 21:08:02.958045 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790284076800039-392-0/minicluster-data/master-0/wal/instance
I20260924 21:08:02.958182 392 external_mini_cluster.cc:1428] Setting key 6578ee1dc07c9689be52c673959cd80f
I20260924 21:08:02.959686 998 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2fedf5e0451b42bebbef2c882bcf2cd6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2fedf5e0451b42bebbef2c882bcf2cd6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35133 } }
I20260924 21:08:02.959830 998 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2fedf5e0451b42bebbef2c882bcf2cd6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 21:08:02.959865 998 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2fedf5e0451b42bebbef2c882bcf2cd6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2fedf5e0451b42bebbef2c882bcf2cd6, State: Initialized, Role: FOLLOWER
I20260924 21:08:02.960048 998 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2fedf5e0451b42bebbef2c882bcf2cd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2fedf5e0451b42bebbef2c882bcf2cd6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35133 } }
I20260924 21:08:02.960150 998 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2fedf5e0451b42bebbef2c882bcf2cd6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 21:08:02.960180 998 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2fedf5e0451b42bebbef2c882bcf2cd6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 21:08:02.960227 998 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2fedf5e0451b42bebbef2c882bcf2cd6 [term 0 FOLLOWER]: Advancing to term 1
I20260924 21:08:02.961061 998 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2fedf5e0451b42bebbef2c882bcf2cd6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2fedf5e0451b42bebbef2c882bcf2cd6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35133 } }
I20260924 21:08:02.961134 998 leader_election.cc:302] T 00000000000000000000000000000000 P 2fedf5e0451b42bebbef2c882bcf2cd6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2fedf5e0451b42bebbef2c882bcf2cd6; no voters:
I20260924 21:08:02.961298 998 leader_election.cc:288] T 00000000000000000000000000000000 P 2fedf5e0451b42bebbef2c882bcf2cd6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 21:08:02.961328 1003 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2fedf5e0451b42bebbef2c882bcf2cd6 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 21:08:02.961449 1003 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2fedf5e0451b42bebbef2c882bcf2cd6 [term 1 LEADER]: Becoming Leader. State: Replica: 2fedf5e0451b42bebbef2c882bcf2cd6, State: Running, Role: LEADER
I20260924 21:08:02.961524 998 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2fedf5e0451b42bebbef2c882bcf2cd6 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 21:08:02.961565 1003 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2fedf5e0451b42bebbef2c882bcf2cd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2fedf5e0451b42bebbef2c882bcf2cd6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35133 } }
I20260924 21:08:02.961887 1003 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2fedf5e0451b42bebbef2c882bcf2cd6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2fedf5e0451b42bebbef2c882bcf2cd6. Latest consensus state: current_term: 1 leader_uuid: "2fedf5e0451b42bebbef2c882bcf2cd6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2fedf5e0451b42bebbef2c882bcf2cd6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35133 } } }
I20260924 21:08:02.961897 1004 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2fedf5e0451b42bebbef2c882bcf2cd6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2fedf5e0451b42bebbef2c882bcf2cd6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2fedf5e0451b42bebbef2c882bcf2cd6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35133 } } }
I20260924 21:08:02.962039 1003 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2fedf5e0451b42bebbef2c882bcf2cd6 [sys.catalog]: This master's current role is: LEADER
I20260924 21:08:02.962128 1004 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2fedf5e0451b42bebbef2c882bcf2cd6 [sys.catalog]: This master's current role is: LEADER
W20260924 21:08:02.963084 1019 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 2fedf5e0451b42bebbef2c882bcf2cd6: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 21:08:02.963153 1019 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 21:08:02.963254 1012 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 21:08:02.963701 1012 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 24 21:08:02 dist-test-slave-c7tk krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284082, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 21:08:02.965704 1012 catalog_manager.cc:1347] Generated new cluster ID: a885c102dea54779bc388f0971cca40e
I20260924 21:08:02.965751 1012 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 21:08:02.968655 1002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:02.958672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34608 (local address 127.0.98.62:35133)
0924 21:08:02.958838 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0924 21:08:02.958845 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 21:08:02.958866 (+ 21us) server_negotiation.cc:407] Connection header received
0924 21:08:02.959674 (+ 808us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:08:02.959679 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:08:02.959719 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:08:02.959817 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:08:02.960564 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:02.961566 (+ 1002us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:02.962518 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:02.963049 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:02.963459 (+ 410us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:02.965663 (+ 2204us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 21:08:02.965666 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 21:08:02.965666 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 21:08:02.965673 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 21:08:02.967217 (+ 1544us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:08:02.967606 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:08:02.967608 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:08:02.967609 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:08:02.967692 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:08:02.967875 (+ 183us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:08:02.967876 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:08:02.967876 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:08:02.968071 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 21:08:02.968254 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0924 21:08:02.968313 (+ 59us) server_negotiation.cc:299] Negotiation successful
0924 21:08:02.968523 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106,"thread_start_us":79,"threads_started":1}
I20260924 21:08:02.982067 1012 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 21:08:02.982477 1012 catalog_manager.cc:1504] Loading token signing keys...
I20260924 21:08:02.986881 1012 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2fedf5e0451b42bebbef2c882bcf2cd6: Generated new TSK 0
I20260924 21:08:02.987046 1012 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790284076800039-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790284076800039-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260924 21:08:03.013000 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAzNNjS/build/release/bin/kudu
/tmp/dist-test-taskAzNNjS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790284076800039-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790284076800039-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790284076800039-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790284076800039-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35133
--keytab_file=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790284076800039-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39391
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790284076800039-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790284076800039-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790284076800039-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 21:08:03.084185 1024 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 21:08:03.084352 1024 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 21:08:03.084375 1024 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 21:08:03.085388 1024 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 21:08:03.085427 1024 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 21:08:03.085474 1024 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 21:08:03.086894 1024 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39391
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790284076800039-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790284076800039-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790284076800039-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790284076800039-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35133
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1024
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790284076800039-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 19:49:06 UTC on ccycloud.quasar-vhqjoc.root.comops.site
build id 4670864
I20260924 21:08:03.087121 1024 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 21:08:03.087342 1024 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 21:08:03.091564 1029 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 21:08:03.091615 1030 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 21:08:03.091679 1024 server_base.cc:1034] running on GCE node
W20260924 21:08:03.091555 1032 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 21:08:03.091907 1024 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 21:08:03.092069 1024 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 21:08:03.093190 1024 hybrid_clock.cc:648] HybridClock initialized: now 1790284083093169 us; error 36 us; skew 500 ppm
Sep 24 21:08:03 dist-test-slave-c7tk krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284083, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 21:08:03.098556 1024 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 21:08:03.098857 1024 webserver.cc:460] Webserver started at http://127.0.98.1:44823/ using document root <none> and password file <none>
I20260924 21:08:03.099050 1024 fs_manager.cc:359] Metadata directory not provided
I20260924 21:08:03.099109 1024 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 21:08:03.099237 1024 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 21:08:03.100100 1024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790284076800039-392-0/minicluster-data/ts-0/data/instance:
uuid: "ffce785959354b13bc6386b081f8f07d"
format_stamp: "Formatted at 2026-09-24 21:08:03 on dist-test-slave-c7tk"
server_key: "68e37afd17b6e9fc7b25e7a43e0a3fa7"
server_key_iv: "a30681cbd41e076e47a2ee472bf1c21e"
server_key_version: "encryptionkey@0"
I20260924 21:08:03.100422 1024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790284076800039-392-0/minicluster-data/ts-0/wal/instance:
uuid: "ffce785959354b13bc6386b081f8f07d"
format_stamp: "Formatted at 2026-09-24 21:08:03 on dist-test-slave-c7tk"
server_key: "68e37afd17b6e9fc7b25e7a43e0a3fa7"
server_key_iv: "a30681cbd41e076e47a2ee472bf1c21e"
server_key_version: "encryptionkey@0"
I20260924 21:08:03.101531 1024 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 21:08:03.102205 1039 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 21:08:03.102362 1024 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 21:08:03.102448 1024 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790284076800039-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790284076800039-392-0/minicluster-data/ts-0/wal
uuid: "ffce785959354b13bc6386b081f8f07d"
format_stamp: "Formatted at 2026-09-24 21:08:03 on dist-test-slave-c7tk"
server_key: "68e37afd17b6e9fc7b25e7a43e0a3fa7"
server_key_iv: "a30681cbd41e076e47a2ee472bf1c21e"
server_key_version: "encryptionkey@0"
I20260924 21:08:03.102525 1024 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790284076800039-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790284076800039-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790284076800039-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 21:08:03.114475 1024 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 21:08:03.116143 1024 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 21:08:03.116272 1024 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 21:08:03.116492 1024 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 21:08:03.116775 1024 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 21:08:03.116830 1024 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 21:08:03.116868 1024 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 21:08:03.116894 1024 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 21:08:03.121724 1024 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:37939
I20260924 21:08:03.121783 1152 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:37939 every 8 connection(s)
I20260924 21:08:03.122081 1024 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790284076800039-392-0/minicluster-data/ts-0/data/info.pb
I20260924 21:08:03.126658 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAzNNjS/build/release/bin/kudu as pid 1024
I20260924 21:08:03.126760 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790284076800039-392-0/minicluster-data/ts-0/wal/instance
I20260924 21:08:03.126919 392 external_mini_cluster.cc:1428] Setting key 42c950d73d9cc3d6510fcd8e1420158d
Sep 24 21:08:03 dist-test-slave-c7tk krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284083, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 21:08:03.130260 1002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:03.122680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:39095 (local address 127.0.98.62:35133)
0924 21:08:03.122716 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0924 21:08:03.122718 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 21:08:03.122913 (+ 195us) server_negotiation.cc:407] Connection header received
0924 21:08:03.123323 (+ 410us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:08:03.123324 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:08:03.123337 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:08:03.123382 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:08:03.124187 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:03.124637 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:03.125501 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:03.125681 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:03.125755 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:03.127845 (+ 2090us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 21:08:03.127846 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 21:08:03.127847 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 21:08:03.127853 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 21:08:03.129064 (+ 1211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:08:03.129409 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:08:03.129410 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:08:03.129410 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:08:03.129471 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:08:03.129683 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:08:03.129684 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:08:03.129684 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:08:03.129779 (+ 95us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 21:08:03.129877 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0924 21:08:03.130102 (+ 225us) server_negotiation.cc:299] Negotiation successful
0924 21:08:03.130182 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260924 21:08:03.130380 1155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:03.122662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35133 (local address 127.0.98.1:39095)
0924 21:08:03.122812 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0924 21:08:03.122838 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0924 21:08:03.123231 (+ 393us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 21:08:03.123438 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 21:08:03.123441 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 21:08:03.123934 (+ 493us) client_negotiation.cc:189] Negotiated authn=SASL
0924 21:08:03.124089 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:03.124093 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:03.124721 (+ 628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:03.124722 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:03.125408 (+ 686us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:03.125409 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:03.125763 (+ 354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:03.125764 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:03.125891 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:03.126273 (+ 382us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 21:08:03.126280 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0924 21:08:03.127747 (+ 1467us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 21:08:03.129148 (+ 1401us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 21:08:03.129153 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 21:08:03.129155 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 21:08:03.129336 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 21:08:03.129527 (+ 191us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 21:08:03.129528 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 21:08:03.129528 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 21:08:03.129625 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 21:08:03.129870 (+ 245us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 21:08:03.129875 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 21:08:03.129991 (+ 116us) client_negotiation.cc:769] Sending connection context
0924 21:08:03.130082 (+ 91us) client_negotiation.cc:240] Negotiation successful
0924 21:08:03.130201 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":72,"threads_started":1}
I20260924 21:08:03.130748 1153 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35133
I20260924 21:08:03.130834 1153 heartbeater.cc:464] Registering TS with master...
I20260924 21:08:03.130997 1153 heartbeater.cc:511] Master 127.0.98.62:35133 requested a full tablet report, sending...
I20260924 21:08:03.131397 962 ts_manager.cc:194] Registered new tserver with Master: ffce785959354b13bc6386b081f8f07d (127.0.98.1:37939)
I20260924 21:08:03.131924 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:39095
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790284076800039-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790284076800039-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260924 21:08:03.148864 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAzNNjS/build/release/bin/kudu
/tmp/dist-test-taskAzNNjS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790284076800039-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790284076800039-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790284076800039-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790284076800039-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35133
--keytab_file=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790284076800039-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39391
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790284076800039-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790284076800039-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790284076800039-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 21:08:03.219300 1158 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 21:08:03.219460 1158 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 21:08:03.219480 1158 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 21:08:03.220424 1158 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 21:08:03.220454 1158 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 21:08:03.220476 1158 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 21:08:03.221606 1158 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39391
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790284076800039-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790284076800039-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790284076800039-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790284076800039-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35133
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1158
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790284076800039-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 19:49:06 UTC on ccycloud.quasar-vhqjoc.root.comops.site
build id 4670864
I20260924 21:08:03.221742 1158 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 21:08:03.221944 1158 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 21:08:03.225986 1166 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 21:08:03.225992 1163 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 21:08:03.226127 1158 server_base.cc:1034] running on GCE node
W20260924 21:08:03.225992 1164 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 21:08:03.226361 1158 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 21:08:03.226604 1158 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 21:08:03.227766 1158 hybrid_clock.cc:648] HybridClock initialized: now 1790284083227760 us; error 38 us; skew 500 ppm
Sep 24 21:08:03 dist-test-slave-c7tk krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284083, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 21:08:03.229961 1158 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 21:08:03.230300 1158 webserver.cc:460] Webserver started at http://127.0.98.2:42459/ using document root <none> and password file <none>
I20260924 21:08:03.230495 1158 fs_manager.cc:359] Metadata directory not provided
I20260924 21:08:03.230551 1158 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 21:08:03.230656 1158 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 21:08:03.231535 1158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790284076800039-392-0/minicluster-data/ts-1/data/instance:
uuid: "7b9ab3ea00a04280abbf366cd04be59e"
format_stamp: "Formatted at 2026-09-24 21:08:03 on dist-test-slave-c7tk"
server_key: "3ba990e0ec5ce972da5f749c1412fa35"
server_key_iv: "4e617da87e986904c7d8688a80761df4"
server_key_version: "encryptionkey@0"
I20260924 21:08:03.231875 1158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790284076800039-392-0/minicluster-data/ts-1/wal/instance:
uuid: "7b9ab3ea00a04280abbf366cd04be59e"
format_stamp: "Formatted at 2026-09-24 21:08:03 on dist-test-slave-c7tk"
server_key: "3ba990e0ec5ce972da5f749c1412fa35"
server_key_iv: "4e617da87e986904c7d8688a80761df4"
server_key_version: "encryptionkey@0"
I20260924 21:08:03.233024 1158 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 21:08:03.233752 1173 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 21:08:03.233935 1158 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 21:08:03.234021 1158 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790284076800039-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790284076800039-392-0/minicluster-data/ts-1/wal
uuid: "7b9ab3ea00a04280abbf366cd04be59e"
format_stamp: "Formatted at 2026-09-24 21:08:03 on dist-test-slave-c7tk"
server_key: "3ba990e0ec5ce972da5f749c1412fa35"
server_key_iv: "4e617da87e986904c7d8688a80761df4"
server_key_version: "encryptionkey@0"
I20260924 21:08:03.234071 1158 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790284076800039-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790284076800039-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790284076800039-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 21:08:03.249509 1158 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 21:08:03.251533 1158 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 21:08:03.251636 1158 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 21:08:03.251770 1158 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 21:08:03.252081 1158 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 21:08:03.252115 1158 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 21:08:03.252135 1158 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 21:08:03.252182 1158 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 21:08:03.257752 1158 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:46037
I20260924 21:08:03.257853 1286 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:46037 every 8 connection(s)
I20260924 21:08:03.258102 1158 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790284076800039-392-0/minicluster-data/ts-1/data/info.pb
I20260924 21:08:03.262635 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAzNNjS/build/release/bin/kudu as pid 1158
I20260924 21:08:03.262714 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790284076800039-392-0/minicluster-data/ts-1/wal/instance
I20260924 21:08:03.262853 392 external_mini_cluster.cc:1428] Setting key 1183bacac676c358f0755eb63e38d01f
Sep 24 21:08:03 dist-test-slave-c7tk krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284083, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 21:08:03.267943 1289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:03.258665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35133 (local address 127.0.98.2:51027)
0924 21:08:03.258796 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0924 21:08:03.258824 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0924 21:08:03.259212 (+ 388us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 21:08:03.259374 (+ 162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 21:08:03.259391 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 21:08:03.260096 (+ 705us) client_negotiation.cc:189] Negotiated authn=SASL
0924 21:08:03.260342 (+ 246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:03.260347 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:03.261119 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:03.261121 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:03.262376 (+ 1255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:03.262379 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:03.262949 (+ 570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:03.262950 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:03.263101 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:03.263575 (+ 474us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 21:08:03.263585 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 21:08:03.264964 (+ 1379us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 21:08:03.266716 (+ 1752us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 21:08:03.266722 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 21:08:03.266724 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 21:08:03.266961 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 21:08:03.267169 (+ 208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 21:08:03.267169 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 21:08:03.267169 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 21:08:03.267256 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 21:08:03.267530 (+ 274us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 21:08:03.267533 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 21:08:03.267638 (+ 105us) client_negotiation.cc:769] Sending connection context
0924 21:08:03.267719 (+ 81us) client_negotiation.cc:240] Negotiation successful
0924 21:08:03.267812 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":79,"threads_started":1}
I20260924 21:08:03.267982 1002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:03.258674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:51027 (local address 127.0.98.62:35133)
0924 21:08:03.258732 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0924 21:08:03.258734 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 21:08:03.258900 (+ 166us) server_negotiation.cc:407] Connection header received
0924 21:08:03.259280 (+ 380us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:08:03.259281 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:08:03.259290 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:08:03.259334 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:08:03.260449 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:03.261047 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:03.262487 (+ 1440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:03.262846 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:03.262916 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:03.265046 (+ 2130us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 21:08:03.265048 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 21:08:03.265048 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 21:08:03.265056 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 21:08:03.266619 (+ 1563us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:08:03.267038 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:08:03.267040 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:08:03.267040 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:08:03.267101 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:08:03.267320 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:08:03.267322 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:08:03.267322 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:08:03.267465 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 21:08:03.267538 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0924 21:08:03.267763 (+ 225us) server_negotiation.cc:299] Negotiation successful
0924 21:08:03.267843 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260924 21:08:03.268343 1287 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35133
I20260924 21:08:03.268435 1287 heartbeater.cc:464] Registering TS with master...
I20260924 21:08:03.268592 1287 heartbeater.cc:511] Master 127.0.98.62:35133 requested a full tablet report, sending...
I20260924 21:08:03.268940 962 ts_manager.cc:194] Registered new tserver with Master: 7b9ab3ea00a04280abbf366cd04be59e (127.0.98.2:46037)
I20260924 21:08:03.269387 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:51027
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790284076800039-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790284076800039-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260924 21:08:03.284106 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAzNNjS/build/release/bin/kudu
/tmp/dist-test-taskAzNNjS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790284076800039-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790284076800039-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790284076800039-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790284076800039-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35133
--keytab_file=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790284076800039-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39391
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790284076800039-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790284076800039-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790284076800039-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 21:08:03.355578 1292 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 21:08:03.355746 1292 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 21:08:03.355765 1292 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 21:08:03.356833 1292 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 21:08:03.356873 1292 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 21:08:03.356902 1292 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 21:08:03.358290 1292 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39391
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790284076800039-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790284076800039-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790284076800039-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790284076800039-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35133
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1292
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790284076800039-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 19:49:06 UTC on ccycloud.quasar-vhqjoc.root.comops.site
build id 4670864
I20260924 21:08:03.358501 1292 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 21:08:03.358692 1292 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 21:08:03.363096 1297 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 21:08:03.363191 1298 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 21:08:03.363131 1292 server_base.cc:1034] running on GCE node
W20260924 21:08:03.363320 1300 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 21:08:03.363502 1292 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 21:08:03.363780 1292 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 21:08:03.364928 1292 hybrid_clock.cc:648] HybridClock initialized: now 1790284083364908 us; error 31 us; skew 500 ppm
Sep 24 21:08:03 dist-test-slave-c7tk krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284083, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 21:08:03.367168 1292 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 21:08:03.367623 1292 webserver.cc:460] Webserver started at http://127.0.98.3:33839/ using document root <none> and password file <none>
I20260924 21:08:03.367871 1292 fs_manager.cc:359] Metadata directory not provided
I20260924 21:08:03.367913 1292 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 21:08:03.368054 1292 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 21:08:03.368952 1292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790284076800039-392-0/minicluster-data/ts-2/data/instance:
uuid: "5bd047bc3a934a9aa7fe04cb8f886aab"
format_stamp: "Formatted at 2026-09-24 21:08:03 on dist-test-slave-c7tk"
server_key: "3acec4ae7c8814a2900f7169d87251ca"
server_key_iv: "442c409945e88d1f60eaa8c50deac355"
server_key_version: "encryptionkey@0"
I20260924 21:08:03.369248 1292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790284076800039-392-0/minicluster-data/ts-2/wal/instance:
uuid: "5bd047bc3a934a9aa7fe04cb8f886aab"
format_stamp: "Formatted at 2026-09-24 21:08:03 on dist-test-slave-c7tk"
server_key: "3acec4ae7c8814a2900f7169d87251ca"
server_key_iv: "442c409945e88d1f60eaa8c50deac355"
server_key_version: "encryptionkey@0"
I20260924 21:08:03.370461 1292 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 21:08:03.371133 1307 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 21:08:03.371296 1292 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 21:08:03.371388 1292 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790284076800039-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790284076800039-392-0/minicluster-data/ts-2/wal
uuid: "5bd047bc3a934a9aa7fe04cb8f886aab"
format_stamp: "Formatted at 2026-09-24 21:08:03 on dist-test-slave-c7tk"
server_key: "3acec4ae7c8814a2900f7169d87251ca"
server_key_iv: "442c409945e88d1f60eaa8c50deac355"
server_key_version: "encryptionkey@0"
I20260924 21:08:03.371456 1292 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790284076800039-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790284076800039-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790284076800039-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 21:08:03.380939 1292 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 21:08:03.382437 1292 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 21:08:03.382565 1292 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 21:08:03.382732 1292 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 21:08:03.382972 1292 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 21:08:03.383018 1292 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 21:08:03.383051 1292 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 21:08:03.383074 1292 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 21:08:03.388559 1292 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:40587
I20260924 21:08:03.388590 1420 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:40587 every 8 connection(s)
I20260924 21:08:03.388890 1292 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790284076800039-392-0/minicluster-data/ts-2/data/info.pb
Sep 24 21:08:03 dist-test-slave-c7tk krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284083, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 21:08:03.396823 1002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:03.389517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:41081 (local address 127.0.98.62:35133)
0924 21:08:03.389554 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0924 21:08:03.389555 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 21:08:03.389720 (+ 165us) server_negotiation.cc:407] Connection header received
0924 21:08:03.390182 (+ 462us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:08:03.390183 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:08:03.390195 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:08:03.390244 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:08:03.391111 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:03.391594 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:03.392461 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:03.392618 (+ 157us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:03.392662 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:03.394401 (+ 1739us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 21:08:03.394403 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 21:08:03.394403 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 21:08:03.394410 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 21:08:03.395617 (+ 1207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:08:03.395977 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:08:03.395978 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:08:03.395978 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:08:03.396029 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:08:03.396234 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:08:03.396234 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:08:03.396235 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:08:03.396335 (+ 100us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 21:08:03.396394 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0924 21:08:03.396664 (+ 270us) server_negotiation.cc:299] Negotiation successful
0924 21:08:03.396736 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260924 21:08:03.396994 1423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:03.389486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35133 (local address 127.0.98.3:41081)
0924 21:08:03.389616 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0924 21:08:03.389650 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0924 21:08:03.390100 (+ 450us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 21:08:03.390295 (+ 195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 21:08:03.390298 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 21:08:03.390857 (+ 559us) client_negotiation.cc:189] Negotiated authn=SASL
0924 21:08:03.391035 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:03.391039 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:03.391666 (+ 627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:03.391667 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:03.392387 (+ 720us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:03.392389 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:03.392681 (+ 292us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:03.392682 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:03.392764 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:03.393105 (+ 341us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 21:08:03.393111 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0924 21:08:03.394312 (+ 1201us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 21:08:03.395706 (+ 1394us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 21:08:03.395711 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 21:08:03.395713 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 21:08:03.395893 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 21:08:03.396084 (+ 191us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 21:08:03.396085 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 21:08:03.396085 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 21:08:03.396177 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 21:08:03.396402 (+ 225us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 21:08:03.396407 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 21:08:03.396540 (+ 133us) client_negotiation.cc:769] Sending connection context
0924 21:08:03.396640 (+ 100us) client_negotiation.cc:240] Negotiation successful
0924 21:08:03.396777 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":73,"threads_started":1}
I20260924 21:08:03.397495 1421 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35133
I20260924 21:08:03.397660 1421 heartbeater.cc:464] Registering TS with master...
I20260924 21:08:03.397719 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAzNNjS/build/release/bin/kudu as pid 1292
I20260924 21:08:03.397836 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790284076800039-392-0/minicluster-data/ts-2/wal/instance
I20260924 21:08:03.397898 1421 heartbeater.cc:511] Master 127.0.98.62:35133 requested a full tablet report, sending...
I20260924 21:08:03.397996 392 external_mini_cluster.cc:1428] Setting key 10e4ee8456a23e88ba255b43f2587be0
I20260924 21:08:03.398320 962 ts_manager.cc:194] Registered new tserver with Master: 5bd047bc3a934a9aa7fe04cb8f886aab (127.0.98.3:40587)
I20260924 21:08:03.398765 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:41081
I20260924 21:08:03.398936 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 21:08:03.405231 1002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:03.400441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34616 (local address 127.0.98.62:35133)
0924 21:08:03.400471 (+ 30us) server_negotiation.cc:206] Beginning negotiation
0924 21:08:03.400472 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 21:08:03.400584 (+ 112us) server_negotiation.cc:407] Connection header received
0924 21:08:03.400603 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:08:03.400604 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:08:03.400611 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:08:03.400655 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:08:03.401098 (+ 443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:03.401491 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:03.402149 (+ 658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:03.402309 (+ 160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:03.402352 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:03.403332 (+ 980us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 21:08:03.403333 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 21:08:03.403333 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 21:08:03.403338 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 21:08:03.404321 (+ 983us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:08:03.404622 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:08:03.404623 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:08:03.404623 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:08:03.404663 (+ 40us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:08:03.404852 (+ 189us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:08:03.404853 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:08:03.404853 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:08:03.404971 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 21:08:03.405037 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0924 21:08:03.405137 (+ 100us) server_negotiation.cc:299] Negotiation successful
0924 21:08:03.405185 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260924 21:08:03.406975 962 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34616:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 21:08:03.408344 962 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 21:08:03.413460 1429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:03.410104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37939 (local address 127.0.0.1:41648)
0924 21:08:03.410226 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0924 21:08:03.410255 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0924 21:08:03.410308 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 21:08:03.410525 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 21:08:03.410528 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 21:08:03.410529 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 21:08:03.410579 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:03.410581 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:03.411597 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:03.411598 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:03.412466 (+ 868us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:03.412468 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:03.413212 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:03.413213 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:03.413275 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:03.413286 (+ 11us) client_negotiation.cc:769] Sending connection context
0924 21:08:03.413359 (+ 73us) client_negotiation.cc:240] Negotiation successful
0924 21:08:03.413399 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":40,"threads_started":1}
I20260924 21:08:03.413483 1432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:03.410208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41648 (local address 127.0.98.1:37939)
0924 21:08:03.410372 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0924 21:08:03.410375 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 21:08:03.410383 (+ 8us) server_negotiation.cc:407] Connection header received
0924 21:08:03.410422 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:08:03.410424 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:08:03.410438 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:08:03.410508 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 21:08:03.410654 (+ 146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:03.411508 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:03.412542 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:03.413147 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:03.413204 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:03.413251 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0924 21:08:03.413372 (+ 121us) server_negotiation.cc:299] Negotiation successful
0924 21:08:03.413416 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":71,"threads_started":1}
I20260924 21:08:03.413909 1430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:03.410104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:46037 (local address 127.0.0.1:44222)
0924 21:08:03.410337 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0924 21:08:03.410347 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 21:08:03.410408 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 21:08:03.410780 (+ 372us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 21:08:03.410781 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 21:08:03.410782 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 21:08:03.410818 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:03.410819 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:03.411878 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:03.411879 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:03.412799 (+ 920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:03.412801 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:03.413673 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:03.413674 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:03.413729 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:03.413733 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 21:08:03.413778 (+ 45us) client_negotiation.cc:240] Negotiation successful
0924 21:08:03.413843 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":219,"mutex_wait_us":5,"thread_start_us":43,"threads_started":1}
I20260924 21:08:03.414009 1433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:03.410246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40587 (local address 127.0.0.1:53526)
0924 21:08:03.410867 (+ 621us) negotiation.cc:107] Waiting for socket to connect
0924 21:08:03.410892 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0924 21:08:03.410933 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 21:08:03.411125 (+ 192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 21:08:03.411126 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 21:08:03.411127 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 21:08:03.411158 (+ 31us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:03.411159 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:03.412117 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:03.412118 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:03.412994 (+ 876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:03.412996 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:03.413775 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:03.413776 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:03.413849 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:03.413853 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 21:08:03.413907 (+ 54us) client_negotiation.cc:240] Negotiation successful
0924 21:08:03.413945 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":607,"mutex_wait_us":2,"thread_start_us":74,"threads_started":1}
I20260924 21:08:03.414009 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:03.410403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53526 (local address 127.0.98.3:40587)
0924 21:08:03.410534 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0924 21:08:03.410537 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 21:08:03.410981 (+ 444us) server_negotiation.cc:407] Connection header received
0924 21:08:03.411012 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:08:03.411014 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:08:03.411024 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:08:03.411079 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 21:08:03.411210 (+ 131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:03.412029 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:03.413054 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:03.413690 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:03.413774 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:03.413838 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0924 21:08:03.413919 (+ 81us) server_negotiation.cc:299] Negotiation successful
0924 21:08:03.413950 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109,"thread_start_us":84,"threads_started":1}
I20260924 21:08:03.414201 1431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:03.410195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44222 (local address 127.0.98.2:46037)
0924 21:08:03.410652 (+ 457us) server_negotiation.cc:206] Beginning negotiation
0924 21:08:03.410655 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 21:08:03.410663 (+ 8us) server_negotiation.cc:407] Connection header received
0924 21:08:03.410699 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:08:03.410700 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:08:03.410710 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:08:03.410782 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 21:08:03.410892 (+ 110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:03.411798 (+ 906us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:03.412879 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:03.413503 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:03.413564 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:03.413628 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0924 21:08:03.414092 (+ 464us) server_negotiation.cc:299] Negotiation successful
0924 21:08:03.414132 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":424,"thread_start_us":68,"threads_started":1}
I20260924 21:08:03.414366 1087 tablet_service.cc:1467] Processing CreateTablet for tablet d6da1b2b6cce4fb2af175bff040dd3b7 (DEFAULT_TABLE table=test-table [id=30f9b568dea54195a172bb6e2b96d444]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 21:08:03.414633 1087 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d6da1b2b6cce4fb2af175bff040dd3b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 21:08:03.414563 1355 tablet_service.cc:1467] Processing CreateTablet for tablet d6da1b2b6cce4fb2af175bff040dd3b7 (DEFAULT_TABLE table=test-table [id=30f9b568dea54195a172bb6e2b96d444]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 21:08:03.414700 1221 tablet_service.cc:1467] Processing CreateTablet for tablet d6da1b2b6cce4fb2af175bff040dd3b7 (DEFAULT_TABLE table=test-table [id=30f9b568dea54195a172bb6e2b96d444]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 21:08:03.414832 1355 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d6da1b2b6cce4fb2af175bff040dd3b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 21:08:03.414947 1221 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d6da1b2b6cce4fb2af175bff040dd3b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 21:08:03.416981 1435 tablet_bootstrap.cc:492] T d6da1b2b6cce4fb2af175bff040dd3b7 P ffce785959354b13bc6386b081f8f07d: Bootstrap starting.
I20260924 21:08:03.417609 1435 tablet_bootstrap.cc:654] T d6da1b2b6cce4fb2af175bff040dd3b7 P ffce785959354b13bc6386b081f8f07d: Neither blocks nor log segments found. Creating new log.
I20260924 21:08:03.418146 1436 tablet_bootstrap.cc:492] T d6da1b2b6cce4fb2af175bff040dd3b7 P 5bd047bc3a934a9aa7fe04cb8f886aab: Bootstrap starting.
I20260924 21:08:03.418152 1435 log.cc:824] T d6da1b2b6cce4fb2af175bff040dd3b7 P ffce785959354b13bc6386b081f8f07d: Log is configured to *not* fsync() on all Append() calls
I20260924 21:08:03.418502 1437 tablet_bootstrap.cc:492] T d6da1b2b6cce4fb2af175bff040dd3b7 P 7b9ab3ea00a04280abbf366cd04be59e: Bootstrap starting.
I20260924 21:08:03.418768 1436 tablet_bootstrap.cc:654] T d6da1b2b6cce4fb2af175bff040dd3b7 P 5bd047bc3a934a9aa7fe04cb8f886aab: Neither blocks nor log segments found. Creating new log.
I20260924 21:08:03.418853 1435 tablet_bootstrap.cc:492] T d6da1b2b6cce4fb2af175bff040dd3b7 P ffce785959354b13bc6386b081f8f07d: No bootstrap required, opened a new log
I20260924 21:08:03.418916 1435 ts_tablet_manager.cc:1397] T d6da1b2b6cce4fb2af175bff040dd3b7 P ffce785959354b13bc6386b081f8f07d: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260924 21:08:03.419081 1437 tablet_bootstrap.cc:654] T d6da1b2b6cce4fb2af175bff040dd3b7 P 7b9ab3ea00a04280abbf366cd04be59e: Neither blocks nor log segments found. Creating new log.
I20260924 21:08:03.419219 1436 log.cc:824] T d6da1b2b6cce4fb2af175bff040dd3b7 P 5bd047bc3a934a9aa7fe04cb8f886aab: Log is configured to *not* fsync() on all Append() calls
I20260924 21:08:03.419552 1437 log.cc:824] T d6da1b2b6cce4fb2af175bff040dd3b7 P 7b9ab3ea00a04280abbf366cd04be59e: Log is configured to *not* fsync() on all Append() calls
I20260924 21:08:03.419826 1436 tablet_bootstrap.cc:492] T d6da1b2b6cce4fb2af175bff040dd3b7 P 5bd047bc3a934a9aa7fe04cb8f886aab: No bootstrap required, opened a new log
I20260924 21:08:03.419889 1436 ts_tablet_manager.cc:1397] T d6da1b2b6cce4fb2af175bff040dd3b7 P 5bd047bc3a934a9aa7fe04cb8f886aab: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 21:08:03.420226 1437 tablet_bootstrap.cc:492] T d6da1b2b6cce4fb2af175bff040dd3b7 P 7b9ab3ea00a04280abbf366cd04be59e: No bootstrap required, opened a new log
I20260924 21:08:03.420288 1437 ts_tablet_manager.cc:1397] T d6da1b2b6cce4fb2af175bff040dd3b7 P 7b9ab3ea00a04280abbf366cd04be59e: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 21:08:03.420773 1435 raft_consensus.cc:348] T d6da1b2b6cce4fb2af175bff040dd3b7 P ffce785959354b13bc6386b081f8f07d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffce785959354b13bc6386b081f8f07d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37939 } } peers { permanent_uuid: "7b9ab3ea00a04280abbf366cd04be59e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46037 } } peers { permanent_uuid: "5bd047bc3a934a9aa7fe04cb8f886aab" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40587 } }
I20260924 21:08:03.420915 1435 raft_consensus.cc:374] T d6da1b2b6cce4fb2af175bff040dd3b7 P ffce785959354b13bc6386b081f8f07d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 21:08:03.420958 1435 raft_consensus.cc:729] T d6da1b2b6cce4fb2af175bff040dd3b7 P ffce785959354b13bc6386b081f8f07d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ffce785959354b13bc6386b081f8f07d, State: Initialized, Role: FOLLOWER
I20260924 21:08:03.421054 1435 consensus_queue.cc:260] T d6da1b2b6cce4fb2af175bff040dd3b7 P ffce785959354b13bc6386b081f8f07d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffce785959354b13bc6386b081f8f07d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37939 } } peers { permanent_uuid: "7b9ab3ea00a04280abbf366cd04be59e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46037 } } peers { permanent_uuid: "5bd047bc3a934a9aa7fe04cb8f886aab" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40587 } }
I20260924 21:08:03.421269 1435 ts_tablet_manager.cc:1428] T d6da1b2b6cce4fb2af175bff040dd3b7 P ffce785959354b13bc6386b081f8f07d: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260924 21:08:03.421294 1153 heartbeater.cc:503] Master 127.0.98.62:35133 was elected leader, sending a full tablet report...
I20260924 21:08:03.421695 1436 raft_consensus.cc:348] T d6da1b2b6cce4fb2af175bff040dd3b7 P 5bd047bc3a934a9aa7fe04cb8f886aab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffce785959354b13bc6386b081f8f07d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37939 } } peers { permanent_uuid: "7b9ab3ea00a04280abbf366cd04be59e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46037 } } peers { permanent_uuid: "5bd047bc3a934a9aa7fe04cb8f886aab" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40587 } }
I20260924 21:08:03.421849 1436 raft_consensus.cc:374] T d6da1b2b6cce4fb2af175bff040dd3b7 P 5bd047bc3a934a9aa7fe04cb8f886aab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 21:08:03.421902 1436 raft_consensus.cc:729] T d6da1b2b6cce4fb2af175bff040dd3b7 P 5bd047bc3a934a9aa7fe04cb8f886aab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5bd047bc3a934a9aa7fe04cb8f886aab, State: Initialized, Role: FOLLOWER
I20260924 21:08:03.422024 1436 consensus_queue.cc:260] T d6da1b2b6cce4fb2af175bff040dd3b7 P 5bd047bc3a934a9aa7fe04cb8f886aab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffce785959354b13bc6386b081f8f07d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37939 } } peers { permanent_uuid: "7b9ab3ea00a04280abbf366cd04be59e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46037 } } peers { permanent_uuid: "5bd047bc3a934a9aa7fe04cb8f886aab" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40587 } }
I20260924 21:08:03.422006 1437 raft_consensus.cc:348] T d6da1b2b6cce4fb2af175bff040dd3b7 P 7b9ab3ea00a04280abbf366cd04be59e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffce785959354b13bc6386b081f8f07d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37939 } } peers { permanent_uuid: "7b9ab3ea00a04280abbf366cd04be59e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46037 } } peers { permanent_uuid: "5bd047bc3a934a9aa7fe04cb8f886aab" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40587 } }
I20260924 21:08:03.422137 1437 raft_consensus.cc:374] T d6da1b2b6cce4fb2af175bff040dd3b7 P 7b9ab3ea00a04280abbf366cd04be59e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 21:08:03.422163 1437 raft_consensus.cc:729] T d6da1b2b6cce4fb2af175bff040dd3b7 P 7b9ab3ea00a04280abbf366cd04be59e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b9ab3ea00a04280abbf366cd04be59e, State: Initialized, Role: FOLLOWER
I20260924 21:08:03.422266 1436 ts_tablet_manager.cc:1428] T d6da1b2b6cce4fb2af175bff040dd3b7 P 5bd047bc3a934a9aa7fe04cb8f886aab: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260924 21:08:03.422250 1437 consensus_queue.cc:260] T d6da1b2b6cce4fb2af175bff040dd3b7 P 7b9ab3ea00a04280abbf366cd04be59e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffce785959354b13bc6386b081f8f07d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37939 } } peers { permanent_uuid: "7b9ab3ea00a04280abbf366cd04be59e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46037 } } peers { permanent_uuid: "5bd047bc3a934a9aa7fe04cb8f886aab" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40587 } }
I20260924 21:08:03.422330 1421 heartbeater.cc:503] Master 127.0.98.62:35133 was elected leader, sending a full tablet report...
I20260924 21:08:03.422473 1437 ts_tablet_manager.cc:1428] T d6da1b2b6cce4fb2af175bff040dd3b7 P 7b9ab3ea00a04280abbf366cd04be59e: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260924 21:08:03.422479 1287 heartbeater.cc:503] Master 127.0.98.62:35133 was elected leader, sending a full tablet report...
I20260924 21:08:03.451673 1443 raft_consensus.cc:482] T d6da1b2b6cce4fb2af175bff040dd3b7 P 7b9ab3ea00a04280abbf366cd04be59e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 21:08:03.451740 1443 raft_consensus.cc:504] T d6da1b2b6cce4fb2af175bff040dd3b7 P 7b9ab3ea00a04280abbf366cd04be59e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffce785959354b13bc6386b081f8f07d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37939 } } peers { permanent_uuid: "7b9ab3ea00a04280abbf366cd04be59e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46037 } } peers { permanent_uuid: "5bd047bc3a934a9aa7fe04cb8f886aab" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40587 } }
I20260924 21:08:03.452059 1443 leader_election.cc:288] T d6da1b2b6cce4fb2af175bff040dd3b7 P 7b9ab3ea00a04280abbf366cd04be59e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ffce785959354b13bc6386b081f8f07d (127.0.98.1:37939), 5bd047bc3a934a9aa7fe04cb8f886aab (127.0.98.3:40587)
I20260924 21:08:03.454645 1432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:03.452203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:34585 (local address 127.0.98.1:37939)
0924 21:08:03.452263 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0924 21:08:03.452265 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 21:08:03.452272 (+ 7us) server_negotiation.cc:407] Connection header received
0924 21:08:03.452324 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:08:03.452324 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:08:03.452333 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:08:03.452379 (+ 46us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 21:08:03.452511 (+ 132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:03.452930 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:03.453825 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:03.454328 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:03.454403 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:03.454412 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0924 21:08:03.454546 (+ 134us) server_negotiation.cc:299] Negotiation successful
0924 21:08:03.454587 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260924 21:08:03.454715 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:03.452252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:36201 (local address 127.0.98.3:40587)
0924 21:08:03.452284 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0924 21:08:03.452285 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 21:08:03.452403 (+ 118us) server_negotiation.cc:407] Connection header received
0924 21:08:03.452454 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:08:03.452455 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:08:03.452463 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:08:03.452524 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 21:08:03.452642 (+ 118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:03.453049 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:03.453951 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:03.454359 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:03.454449 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:03.454457 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0924 21:08:03.454623 (+ 166us) server_negotiation.cc:299] Negotiation successful
0924 21:08:03.454656 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260924 21:08:03.454727 1289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:03.452174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37939 (local address 127.0.98.2:34585)
0924 21:08:03.452224 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0924 21:08:03.452244 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 21:08:03.452277 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 21:08:03.452398 (+ 121us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 21:08:03.452399 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 21:08:03.452400 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 21:08:03.452449 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:03.452450 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:03.453008 (+ 558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:03.453009 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:03.453718 (+ 709us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:03.453719 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:03.454404 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:03.454405 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:03.454474 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:03.454477 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 21:08:03.454531 (+ 54us) client_negotiation.cc:240] Negotiation successful
0924 21:08:03.454638 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35,"mutex_wait_us":35}
I20260924 21:08:03.454913 1444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:03.452191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40587 (local address 127.0.98.2:36201)
0924 21:08:03.452329 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0924 21:08:03.452340 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 21:08:03.452410 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 21:08:03.452527 (+ 117us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 21:08:03.452528 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 21:08:03.452529 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 21:08:03.452586 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:03.452587 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:03.453121 (+ 534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:03.453122 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:03.453873 (+ 751us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:03.453880 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:03.454456 (+ 576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:03.454457 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:03.454519 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:03.454522 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 21:08:03.454791 (+ 269us) client_negotiation.cc:240] Negotiation successful
0924 21:08:03.454836 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":84,"threads_started":1}
I20260924 21:08:03.455036 1107 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6da1b2b6cce4fb2af175bff040dd3b7" candidate_uuid: "7b9ab3ea00a04280abbf366cd04be59e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ffce785959354b13bc6386b081f8f07d" is_pre_election: true
I20260924 21:08:03.455154 1107 raft_consensus.cc:2454] T d6da1b2b6cce4fb2af175bff040dd3b7 P ffce785959354b13bc6386b081f8f07d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7b9ab3ea00a04280abbf366cd04be59e in term 0.
I20260924 21:08:03.455238 1374 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6da1b2b6cce4fb2af175bff040dd3b7" candidate_uuid: "7b9ab3ea00a04280abbf366cd04be59e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5bd047bc3a934a9aa7fe04cb8f886aab" is_pre_election: true
I20260924 21:08:03.455343 1374 raft_consensus.cc:2454] T d6da1b2b6cce4fb2af175bff040dd3b7 P 5bd047bc3a934a9aa7fe04cb8f886aab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7b9ab3ea00a04280abbf366cd04be59e in term 0.
I20260924 21:08:03.455338 1175 leader_election.cc:302] T d6da1b2b6cce4fb2af175bff040dd3b7 P 7b9ab3ea00a04280abbf366cd04be59e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7b9ab3ea00a04280abbf366cd04be59e, ffce785959354b13bc6386b081f8f07d; no voters:
I20260924 21:08:03.455426 1443 raft_consensus.cc:2781] T d6da1b2b6cce4fb2af175bff040dd3b7 P 7b9ab3ea00a04280abbf366cd04be59e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 21:08:03.455475 1443 raft_consensus.cc:482] T d6da1b2b6cce4fb2af175bff040dd3b7 P 7b9ab3ea00a04280abbf366cd04be59e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 21:08:03.455530 1443 raft_consensus.cc:3037] T d6da1b2b6cce4fb2af175bff040dd3b7 P 7b9ab3ea00a04280abbf366cd04be59e [term 0 FOLLOWER]: Advancing to term 1
I20260924 21:08:03.456030 1443 raft_consensus.cc:504] T d6da1b2b6cce4fb2af175bff040dd3b7 P 7b9ab3ea00a04280abbf366cd04be59e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffce785959354b13bc6386b081f8f07d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37939 } } peers { permanent_uuid: "7b9ab3ea00a04280abbf366cd04be59e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46037 } } peers { permanent_uuid: "5bd047bc3a934a9aa7fe04cb8f886aab" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40587 } }
I20260924 21:08:03.456137 1443 leader_election.cc:288] T d6da1b2b6cce4fb2af175bff040dd3b7 P 7b9ab3ea00a04280abbf366cd04be59e [CANDIDATE]: Term 1 election: Requested vote from peers ffce785959354b13bc6386b081f8f07d (127.0.98.1:37939), 5bd047bc3a934a9aa7fe04cb8f886aab (127.0.98.3:40587)
I20260924 21:08:03.456272 1107 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6da1b2b6cce4fb2af175bff040dd3b7" candidate_uuid: "7b9ab3ea00a04280abbf366cd04be59e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ffce785959354b13bc6386b081f8f07d"
I20260924 21:08:03.456272 1374 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6da1b2b6cce4fb2af175bff040dd3b7" candidate_uuid: "7b9ab3ea00a04280abbf366cd04be59e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5bd047bc3a934a9aa7fe04cb8f886aab"
I20260924 21:08:03.456328 1374 raft_consensus.cc:3037] T d6da1b2b6cce4fb2af175bff040dd3b7 P 5bd047bc3a934a9aa7fe04cb8f886aab [term 0 FOLLOWER]: Advancing to term 1
I20260924 21:08:03.456328 1107 raft_consensus.cc:3037] T d6da1b2b6cce4fb2af175bff040dd3b7 P ffce785959354b13bc6386b081f8f07d [term 0 FOLLOWER]: Advancing to term 1
I20260924 21:08:03.457129 1374 raft_consensus.cc:2454] T d6da1b2b6cce4fb2af175bff040dd3b7 P 5bd047bc3a934a9aa7fe04cb8f886aab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7b9ab3ea00a04280abbf366cd04be59e in term 1.
I20260924 21:08:03.457129 1107 raft_consensus.cc:2454] T d6da1b2b6cce4fb2af175bff040dd3b7 P ffce785959354b13bc6386b081f8f07d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7b9ab3ea00a04280abbf366cd04be59e in term 1.
I20260924 21:08:03.457281 1175 leader_election.cc:302] T d6da1b2b6cce4fb2af175bff040dd3b7 P 7b9ab3ea00a04280abbf366cd04be59e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7b9ab3ea00a04280abbf366cd04be59e, ffce785959354b13bc6386b081f8f07d; no voters:
I20260924 21:08:03.457360 1443 raft_consensus.cc:2781] T d6da1b2b6cce4fb2af175bff040dd3b7 P 7b9ab3ea00a04280abbf366cd04be59e [term 1 FOLLOWER]: Leader election won for term 1
I20260924 21:08:03.457477 1443 raft_consensus.cc:686] T d6da1b2b6cce4fb2af175bff040dd3b7 P 7b9ab3ea00a04280abbf366cd04be59e [term 1 LEADER]: Becoming Leader. State: Replica: 7b9ab3ea00a04280abbf366cd04be59e, State: Running, Role: LEADER
I20260924 21:08:03.457660 1443 consensus_queue.cc:237] T d6da1b2b6cce4fb2af175bff040dd3b7 P 7b9ab3ea00a04280abbf366cd04be59e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffce785959354b13bc6386b081f8f07d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37939 } } peers { permanent_uuid: "7b9ab3ea00a04280abbf366cd04be59e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46037 } } peers { permanent_uuid: "5bd047bc3a934a9aa7fe04cb8f886aab" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40587 } }
I20260924 21:08:03.458359 962 catalog_manager.cc:5495] T d6da1b2b6cce4fb2af175bff040dd3b7 P 7b9ab3ea00a04280abbf366cd04be59e reported cstate change: term changed from 0 to 1, leader changed from <none> to 7b9ab3ea00a04280abbf366cd04be59e (127.0.98.2). New cstate: current_term: 1 leader_uuid: "7b9ab3ea00a04280abbf366cd04be59e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffce785959354b13bc6386b081f8f07d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37939 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7b9ab3ea00a04280abbf366cd04be59e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46037 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5bd047bc3a934a9aa7fe04cb8f886aab" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40587 } health_report { overall_health: UNKNOWN } } }
Sep 24 21:08:03 dist-test-slave-c7tk krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284082, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260924 21:08:03.476368 1432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:03.470758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41650 (local address 127.0.98.1:37939)
0924 21:08:03.470803 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0924 21:08:03.470804 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 21:08:03.470915 (+ 111us) server_negotiation.cc:407] Connection header received
0924 21:08:03.470944 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:08:03.470945 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:08:03.470952 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:08:03.470991 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:08:03.471380 (+ 389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:03.471753 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:03.472291 (+ 538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:03.472429 (+ 138us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:03.472468 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:03.474439 (+ 1971us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 21:08:03.474445 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 21:08:03.474445 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 21:08:03.474450 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 21:08:03.475647 (+ 1197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:08:03.475901 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:08:03.475903 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:08:03.475903 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:08:03.475947 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:08:03.476067 (+ 120us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:08:03.476068 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:08:03.476068 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:08:03.476153 (+ 85us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 21:08:03.476233 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0924 21:08:03.476277 (+ 44us) server_negotiation.cc:299] Negotiation successful
0924 21:08:03.476323 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
Sep 24 21:08:03 dist-test-slave-c7tk krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284083, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 24 21:08:03 dist-test-slave-c7tk krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284083, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260924 21:08:03.495298 1432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:03.489592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41666 (local address 127.0.98.1:37939)
0924 21:08:03.489650 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0924 21:08:03.489651 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 21:08:03.489736 (+ 85us) server_negotiation.cc:407] Connection header received
0924 21:08:03.489813 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:08:03.489813 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:08:03.489821 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:08:03.489884 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:08:03.490413 (+ 529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:03.490796 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:03.491392 (+ 596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:03.491544 (+ 152us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:03.491615 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:03.493254 (+ 1639us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 21:08:03.493255 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 21:08:03.493255 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 21:08:03.493261 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 21:08:03.494275 (+ 1014us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:08:03.494610 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:08:03.494611 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:08:03.494611 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:08:03.494661 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:08:03.494836 (+ 175us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:08:03.494837 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:08:03.494838 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:08:03.494933 (+ 95us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 21:08:03.495019 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0924 21:08:03.495163 (+ 144us) server_negotiation.cc:299] Negotiation successful
0924 21:08:03.495220 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
Sep 24 21:08:03 dist-test-slave-c7tk krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284083, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
W20260924 21:08:03.508553 1288 tablet.cc:2369] T d6da1b2b6cce4fb2af175bff040dd3b7 P 7b9ab3ea00a04280abbf366cd04be59e: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 24 21:08:03 dist-test-slave-c7tk krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284083, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260924 21:08:03.513195 1432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:03.507691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41678 (local address 127.0.98.1:37939)
0924 21:08:03.507773 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0924 21:08:03.507775 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 21:08:03.507812 (+ 37us) server_negotiation.cc:407] Connection header received
0924 21:08:03.507858 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:08:03.507858 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:08:03.507865 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:08:03.507923 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:08:03.508412 (+ 489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:03.508836 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:03.509445 (+ 609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:03.509590 (+ 145us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:03.509631 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:03.511449 (+ 1818us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 21:08:03.511450 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 21:08:03.511451 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 21:08:03.511461 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0924 21:08:03.512452 (+ 991us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:08:03.512738 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:08:03.512739 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:08:03.512739 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:08:03.512778 (+ 39us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:08:03.512910 (+ 132us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:08:03.512910 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:08:03.512910 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:08:03.512973 (+ 63us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 21:08:03.513028 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0924 21:08:03.513107 (+ 79us) server_negotiation.cc:299] Negotiation successful
0924 21:08:03.513148 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
W20260924 21:08:03.513577 1067 tablet_service.cc:2908] Rejecting scan request for tablet d6da1b2b6cce4fb2af175bff040dd3b7: Uninitialized: clean time has not yet been initialized
I20260924 21:08:03.514261 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAzNNjS/build/release/bin/kudu with pid 1024
I20260924 21:08:03.518738 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAzNNjS/build/release/bin/kudu with pid 1158
I20260924 21:08:03.522518 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAzNNjS/build/release/bin/kudu with pid 1292
I20260924 21:08:03.526427 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAzNNjS/build/release/bin/kudu with pid 930
2026-09-24T21:08:03Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (1730 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790284076800039-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 21:08:03 dist-test-slave-c7tk krb5kdc[1459](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 21:08:03 dist-test-slave-c7tk krb5kdc[1459](info): set up 2 sockets
Sep 24 21:08:03 dist-test-slave-c7tk krb5kdc[1459](info): commencing operation
krb5kdc: starting...
W20260924 21:08:04.461200 392 mini_kdc.cc:121] Time spent starting KDC: real 0.915s user 0.002s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790284076800039-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790284076800039-392-0/krb5kdc/test-user.keytab.
Sep 24 21:08:04 dist-test-slave-c7tk krb5kdc[1459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284084, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T21:08:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T21:08:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790284076800039-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790284076800039-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 21:08:04.564101 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAzNNjS/build/release/bin/kudu
/tmp/dist-test-taskAzNNjS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790284076800039-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790284076800039-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790284076800039-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790284076800039-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39063
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790284076800039-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38369
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39063
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790284076800039-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790284076800039-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790284076800039-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 21:08:04.633503 1472 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 21:08:04.633744 1472 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 21:08:04.633781 1472 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 21:08:04.634917 1472 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 21:08:04.634986 1472 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260924 21:08:04.635005 1472 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 21:08:04.635039 1472 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 21:08:04.635061 1472 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 21:08:04.635087 1472 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 21:08:04.636327 1472 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38369
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790284076800039-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790284076800039-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39063
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790284076800039-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39063
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790284076800039-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1472
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790284076800039-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 19:49:06 UTC on ccycloud.quasar-vhqjoc.root.comops.site
build id 4670864
I20260924 21:08:04.636579 1472 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 21:08:04.636798 1472 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 21:08:04.640973 1477 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 21:08:04.641094 1480 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 21:08:04.641134 1472 server_base.cc:1034] running on GCE node
W20260924 21:08:04.641244 1478 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 21:08:04.641484 1472 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 21:08:04.641779 1472 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 21:08:04.642949 1472 hybrid_clock.cc:648] HybridClock initialized: now 1790284084642931 us; error 39 us; skew 500 ppm
Sep 24 21:08:04 dist-test-slave-c7tk krb5kdc[1459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284084, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 21:08:04.644984 1472 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 21:08:04.645342 1472 webserver.cc:460] Webserver started at http://127.0.98.62:38127/ using document root <none> and password file <none>
I20260924 21:08:04.645490 1472 fs_manager.cc:359] Metadata directory not provided
I20260924 21:08:04.645529 1472 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 21:08:04.645601 1472 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 21:08:04.646490 1472 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790284076800039-392-0/minicluster-data/master-0/data/instance:
uuid: "ecbe9754db8046bea8afe31efa3a107a"
format_stamp: "Formatted at 2026-09-24 21:08:04 on dist-test-slave-c7tk"
server_key: "90e4763ff5e64a3d1892f84dabcbb3bb"
server_key_iv: "19bf0b228dc8423acf5ac6d0f976c1d4"
server_key_version: "encryptionkey@0"
I20260924 21:08:04.646801 1472 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790284076800039-392-0/minicluster-data/master-0/wal/instance:
uuid: "ecbe9754db8046bea8afe31efa3a107a"
format_stamp: "Formatted at 2026-09-24 21:08:04 on dist-test-slave-c7tk"
server_key: "90e4763ff5e64a3d1892f84dabcbb3bb"
server_key_iv: "19bf0b228dc8423acf5ac6d0f976c1d4"
server_key_version: "encryptionkey@0"
I20260924 21:08:04.648073 1472 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260924 21:08:04.648664 1487 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 21:08:04.648816 1472 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 21:08:04.648907 1472 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790284076800039-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790284076800039-392-0/minicluster-data/master-0/wal
uuid: "ecbe9754db8046bea8afe31efa3a107a"
format_stamp: "Formatted at 2026-09-24 21:08:04 on dist-test-slave-c7tk"
server_key: "90e4763ff5e64a3d1892f84dabcbb3bb"
server_key_iv: "19bf0b228dc8423acf5ac6d0f976c1d4"
server_key_version: "encryptionkey@0"
I20260924 21:08:04.648959 1472 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790284076800039-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790284076800039-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790284076800039-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 21:08:04.658679 1472 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 21:08:04.660576 1472 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 21:08:04.660696 1472 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 21:08:04.664690 1472 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39063
I20260924 21:08:04.664744 1549 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39063 every 8 connection(s)
I20260924 21:08:04.665037 1472 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790284076800039-392-0/minicluster-data/master-0/data/info.pb
I20260924 21:08:04.665616 1550 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 21:08:04.667627 1550 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ecbe9754db8046bea8afe31efa3a107a: Bootstrap starting.
I20260924 21:08:04.667645 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAzNNjS/build/release/bin/kudu as pid 1472
I20260924 21:08:04.667726 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790284076800039-392-0/minicluster-data/master-0/wal/instance
I20260924 21:08:04.667879 392 external_mini_cluster.cc:1428] Setting key bace5c15dfcc601732b8d26781e19991
I20260924 21:08:04.668280 1550 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ecbe9754db8046bea8afe31efa3a107a: Neither blocks nor log segments found. Creating new log.
I20260924 21:08:04.668594 1550 log.cc:824] T 00000000000000000000000000000000 P ecbe9754db8046bea8afe31efa3a107a: Log is configured to *not* fsync() on all Append() calls
I20260924 21:08:04.669179 1550 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ecbe9754db8046bea8afe31efa3a107a: No bootstrap required, opened a new log
I20260924 21:08:04.670425 1550 raft_consensus.cc:348] T 00000000000000000000000000000000 P ecbe9754db8046bea8afe31efa3a107a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ecbe9754db8046bea8afe31efa3a107a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39063 } }
I20260924 21:08:04.670567 1550 raft_consensus.cc:374] T 00000000000000000000000000000000 P ecbe9754db8046bea8afe31efa3a107a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 21:08:04.670625 1550 raft_consensus.cc:729] T 00000000000000000000000000000000 P ecbe9754db8046bea8afe31efa3a107a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ecbe9754db8046bea8afe31efa3a107a, State: Initialized, Role: FOLLOWER
I20260924 21:08:04.670760 1550 consensus_queue.cc:260] T 00000000000000000000000000000000 P ecbe9754db8046bea8afe31efa3a107a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ecbe9754db8046bea8afe31efa3a107a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39063 } }
I20260924 21:08:04.670848 1550 raft_consensus.cc:388] T 00000000000000000000000000000000 P ecbe9754db8046bea8afe31efa3a107a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 21:08:04.670887 1550 raft_consensus.cc:482] T 00000000000000000000000000000000 P ecbe9754db8046bea8afe31efa3a107a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 21:08:04.670931 1550 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ecbe9754db8046bea8afe31efa3a107a [term 0 FOLLOWER]: Advancing to term 1
I20260924 21:08:04.671370 1550 raft_consensus.cc:504] T 00000000000000000000000000000000 P ecbe9754db8046bea8afe31efa3a107a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ecbe9754db8046bea8afe31efa3a107a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39063 } }
I20260924 21:08:04.671473 1550 leader_election.cc:302] T 00000000000000000000000000000000 P ecbe9754db8046bea8afe31efa3a107a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ecbe9754db8046bea8afe31efa3a107a; no voters:
I20260924 21:08:04.671620 1550 leader_election.cc:288] T 00000000000000000000000000000000 P ecbe9754db8046bea8afe31efa3a107a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 21:08:04.671643 1555 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ecbe9754db8046bea8afe31efa3a107a [term 1 FOLLOWER]: Leader election won for term 1
I20260924 21:08:04.671756 1555 raft_consensus.cc:686] T 00000000000000000000000000000000 P ecbe9754db8046bea8afe31efa3a107a [term 1 LEADER]: Becoming Leader. State: Replica: ecbe9754db8046bea8afe31efa3a107a, State: Running, Role: LEADER
I20260924 21:08:04.671844 1550 sys_catalog.cc:564] T 00000000000000000000000000000000 P ecbe9754db8046bea8afe31efa3a107a [sys.catalog]: configured and running, proceeding with master startup.
I20260924 21:08:04.671911 1555 consensus_queue.cc:237] T 00000000000000000000000000000000 P ecbe9754db8046bea8afe31efa3a107a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ecbe9754db8046bea8afe31efa3a107a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39063 } }
I20260924 21:08:04.672246 1555 sys_catalog.cc:455] T 00000000000000000000000000000000 P ecbe9754db8046bea8afe31efa3a107a [sys.catalog]: SysCatalogTable state changed. Reason: New leader ecbe9754db8046bea8afe31efa3a107a. Latest consensus state: current_term: 1 leader_uuid: "ecbe9754db8046bea8afe31efa3a107a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ecbe9754db8046bea8afe31efa3a107a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39063 } } }
I20260924 21:08:04.672348 1555 sys_catalog.cc:458] T 00000000000000000000000000000000 P ecbe9754db8046bea8afe31efa3a107a [sys.catalog]: This master's current role is: LEADER
I20260924 21:08:04.673208 1557 sys_catalog.cc:455] T 00000000000000000000000000000000 P ecbe9754db8046bea8afe31efa3a107a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ecbe9754db8046bea8afe31efa3a107a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ecbe9754db8046bea8afe31efa3a107a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39063 } } }
I20260924 21:08:04.674496 1557 sys_catalog.cc:458] T 00000000000000000000000000000000 P ecbe9754db8046bea8afe31efa3a107a [sys.catalog]: This master's current role is: LEADER
W20260924 21:08:04.673631 1571 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ecbe9754db8046bea8afe31efa3a107a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 21:08:04.674603 1571 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 21:08:04.674690 1561 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 21:08:04.674961 1561 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 24 21:08:04 dist-test-slave-c7tk krb5kdc[1459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284084, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 21:08:04.676483 1561 catalog_manager.cc:1347] Generated new cluster ID: 6f74edb7b54042a5927dd05c7229af87
I20260924 21:08:04.676533 1561 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 21:08:04.680498 1553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:04.668347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41018 (local address 127.0.98.62:39063)
0924 21:08:04.668521 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0924 21:08:04.668529 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 21:08:04.668562 (+ 33us) server_negotiation.cc:407] Connection header received
0924 21:08:04.668967 (+ 405us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:08:04.668973 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:08:04.669013 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:08:04.669171 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:08:04.669764 (+ 593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:04.671064 (+ 1300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:04.671979 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:04.672517 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:04.672620 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:04.675661 (+ 3041us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 21:08:04.675665 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 21:08:04.675665 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 21:08:04.675676 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 21:08:04.679020 (+ 3344us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:08:04.679381 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:08:04.679383 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:08:04.679383 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:08:04.679468 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:08:04.679661 (+ 193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:08:04.679662 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:08:04.679662 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:08:04.679869 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 21:08:04.680045 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0924 21:08:04.680145 (+ 100us) server_negotiation.cc:299] Negotiation successful
0924 21:08:04.680379 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":83,"threads_started":1}
I20260924 21:08:04.694919 1561 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 21:08:04.695472 1561 catalog_manager.cc:1504] Loading token signing keys...
I20260924 21:08:04.704998 1561 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ecbe9754db8046bea8afe31efa3a107a: Generated new TSK 0
I20260924 21:08:04.705152 1561 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790284076800039-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790284076800039-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260924 21:08:04.730686 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAzNNjS/build/release/bin/kudu
/tmp/dist-test-taskAzNNjS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790284076800039-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790284076800039-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790284076800039-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790284076800039-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39063
--keytab_file=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790284076800039-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38369
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790284076800039-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790284076800039-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790284076800039-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 21:08:04.817484 1576 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 21:08:04.817689 1576 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 21:08:04.817742 1576 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 21:08:04.818768 1576 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 21:08:04.818828 1576 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 21:08:04.818861 1576 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 21:08:04.818894 1576 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 21:08:04.820120 1576 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38369
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790284076800039-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790284076800039-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790284076800039-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790284076800039-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:39063
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1576
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790284076800039-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 19:49:06 UTC on ccycloud.quasar-vhqjoc.root.comops.site
build id 4670864
I20260924 21:08:04.820339 1576 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 21:08:04.820533 1576 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 21:08:04.824572 1584 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 21:08:04.824576 1581 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 21:08:04.824602 1582 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 21:08:04.824923 1576 server_base.cc:1034] running on GCE node
I20260924 21:08:04.825086 1576 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 21:08:04.825270 1576 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 21:08:04.826399 1576 hybrid_clock.cc:648] HybridClock initialized: now 1790284084826361 us; error 53 us; skew 500 ppm
Sep 24 21:08:04 dist-test-slave-c7tk krb5kdc[1459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284084, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 21:08:04.828926 1576 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 21:08:04.829367 1576 webserver.cc:460] Webserver started at http://127.0.98.1:46099/ using document root <none> and password file <none>
I20260924 21:08:04.829571 1576 fs_manager.cc:359] Metadata directory not provided
I20260924 21:08:04.829634 1576 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 21:08:04.829775 1576 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 21:08:04.830717 1576 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790284076800039-392-0/minicluster-data/ts-0/data/instance:
uuid: "c7395ba0e4ad43ea965a4a9847899e4e"
format_stamp: "Formatted at 2026-09-24 21:08:04 on dist-test-slave-c7tk"
server_key: "d0a99d76ee68061f0c413db051440202"
server_key_iv: "f31ce9f5dda3c5bd5059f05f135a0b44"
server_key_version: "encryptionkey@0"
I20260924 21:08:04.831058 1576 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790284076800039-392-0/minicluster-data/ts-0/wal/instance:
uuid: "c7395ba0e4ad43ea965a4a9847899e4e"
format_stamp: "Formatted at 2026-09-24 21:08:04 on dist-test-slave-c7tk"
server_key: "d0a99d76ee68061f0c413db051440202"
server_key_iv: "f31ce9f5dda3c5bd5059f05f135a0b44"
server_key_version: "encryptionkey@0"
I20260924 21:08:04.832273 1576 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260924 21:08:04.832945 1591 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 21:08:04.833102 1576 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 21:08:04.833184 1576 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790284076800039-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790284076800039-392-0/minicluster-data/ts-0/wal
uuid: "c7395ba0e4ad43ea965a4a9847899e4e"
format_stamp: "Formatted at 2026-09-24 21:08:04 on dist-test-slave-c7tk"
server_key: "d0a99d76ee68061f0c413db051440202"
server_key_iv: "f31ce9f5dda3c5bd5059f05f135a0b44"
server_key_version: "encryptionkey@0"
I20260924 21:08:04.833252 1576 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790284076800039-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790284076800039-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790284076800039-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 21:08:04.851111 1576 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 21:08:04.852952 1576 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 21:08:04.853094 1576 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 21:08:04.853631 1576 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 21:08:04.853678 1576 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 21:08:04.853703 1576 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 21:08:04.853718 1576 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 21:08:04 dist-test-slave-c7tk krb5kdc[1459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284084, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 21:08:04.860285 1576 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:43729
I20260924 21:08:04.860347 1706 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:43729 every 8 connection(s)
I20260924 21:08:04.860610 1576 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790284076800039-392-0/minicluster-data/ts-0/data/info.pb
I20260924 21:08:04.862172 1553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:04.853932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:55995 (local address 127.0.98.62:39063)
0924 21:08:04.853989 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0924 21:08:04.853991 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 21:08:04.854242 (+ 251us) server_negotiation.cc:407] Connection header received
0924 21:08:04.854897 (+ 655us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:08:04.854899 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:08:04.854908 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:08:04.854982 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:08:04.855551 (+ 569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:04.856029 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:04.856945 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:04.857191 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:04.857256 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:04.859718 (+ 2462us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 21:08:04.859719 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 21:08:04.859720 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 21:08:04.859726 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 21:08:04.861117 (+ 1391us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:08:04.861482 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:08:04.861483 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:08:04.861483 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:08:04.861527 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:08:04.861680 (+ 153us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:08:04.861681 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:08:04.861681 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:08:04.861763 (+ 82us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 21:08:04.861846 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0924 21:08:04.862058 (+ 212us) server_negotiation.cc:299] Negotiation successful
0924 21:08:04.862118 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260924 21:08:04.862301 1600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:04.853967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39063 (local address 127.0.98.1:55995)
0924 21:08:04.854092 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0924 21:08:04.854131 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0924 21:08:04.854778 (+ 647us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 21:08:04.855049 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 21:08:04.855052 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 21:08:04.855308 (+ 256us) client_negotiation.cc:189] Negotiated authn=SASL
0924 21:08:04.855461 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:04.855465 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:04.856110 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:04.856111 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:04.856836 (+ 725us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:04.856838 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:04.857316 (+ 478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:04.857317 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:04.857418 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:04.857989 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 21:08:04.857998 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 21:08:04.859597 (+ 1599us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 21:08:04.861179 (+ 1582us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 21:08:04.861181 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 21:08:04.861183 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 21:08:04.861427 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 21:08:04.861565 (+ 138us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 21:08:04.861565 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 21:08:04.861566 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 21:08:04.861638 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 21:08:04.861848 (+ 210us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 21:08:04.861851 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 21:08:04.861965 (+ 114us) client_negotiation.cc:769] Sending connection context
0924 21:08:04.862035 (+ 70us) client_negotiation.cc:240] Negotiation successful
0924 21:08:04.862126 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":90,"thread_start_us":63,"threads_started":1}
I20260924 21:08:04.862699 1707 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39063
I20260924 21:08:04.862771 1707 heartbeater.cc:464] Registering TS with master...
I20260924 21:08:04.862924 1707 heartbeater.cc:511] Master 127.0.98.62:39063 requested a full tablet report, sending...
I20260924 21:08:04.863426 1503 ts_manager.cc:194] Registered new tserver with Master: c7395ba0e4ad43ea965a4a9847899e4e (127.0.98.1:43729)
I20260924 21:08:04.864180 1503 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:55995
I20260924 21:08:04.864638 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAzNNjS/build/release/bin/kudu as pid 1576
I20260924 21:08:04.864730 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790284076800039-392-0/minicluster-data/ts-0/wal/instance
I20260924 21:08:04.864869 392 external_mini_cluster.cc:1428] Setting key fa83b75cc4422c35266b179a7b6e2828
I20260924 21:08:04.868810 1553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:04.863592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41185 (local address 127.0.98.62:39063)
0924 21:08:04.863628 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0924 21:08:04.863630 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 21:08:04.863739 (+ 109us) server_negotiation.cc:407] Connection header received
0924 21:08:04.863758 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:08:04.863759 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:08:04.863767 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:08:04.863826 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:08:04.864057 (+ 231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:04.864627 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:04.865522 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:04.865674 (+ 152us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:04.865721 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:04.866568 (+ 847us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 21:08:04.866569 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 21:08:04.866569 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 21:08:04.866574 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 21:08:04.867876 (+ 1302us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:08:04.868210 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:08:04.868211 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:08:04.868211 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:08:04.868248 (+ 37us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:08:04.868424 (+ 176us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:08:04.868424 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:08:04.868424 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:08:04.868496 (+ 72us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 21:08:04.868536 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0924 21:08:04.868704 (+ 168us) server_negotiation.cc:299] Negotiation successful
0924 21:08:04.868766 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260924 21:08:04.868880 1713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:04.863558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39063 (local address 127.0.98.1:41185)
0924 21:08:04.863672 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0924 21:08:04.863680 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0924 21:08:04.863730 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 21:08:04.863829 (+ 99us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 21:08:04.863829 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 21:08:04.863964 (+ 135us) client_negotiation.cc:189] Negotiated authn=SASL
0924 21:08:04.864004 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:04.864005 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:04.864719 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:04.864720 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:04.865378 (+ 658us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:04.865380 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:04.865741 (+ 361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:04.865743 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:04.865824 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:04.866150 (+ 326us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 21:08:04.866154 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 21:08:04.866494 (+ 340us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 21:08:04.867955 (+ 1461us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 21:08:04.867956 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 21:08:04.867957 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 21:08:04.868146 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 21:08:04.868303 (+ 157us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 21:08:04.868303 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 21:08:04.868304 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 21:08:04.868372 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 21:08:04.868551 (+ 179us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 21:08:04.868551 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 21:08:04.868591 (+ 40us) client_negotiation.cc:769] Sending connection context
0924 21:08:04.868662 (+ 71us) client_negotiation.cc:240] Negotiation successful
0924 21:08:04.868749 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":93,"thread_start_us":74,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790284076800039-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790284076800039-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260924 21:08:04.888026 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAzNNjS/build/release/bin/kudu
/tmp/dist-test-taskAzNNjS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790284076800039-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790284076800039-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790284076800039-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790284076800039-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39063
--keytab_file=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790284076800039-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38369
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790284076800039-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790284076800039-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790284076800039-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 21:08:04.958012 1716 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 21:08:04.958196 1716 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 21:08:04.958232 1716 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 21:08:04.959220 1716 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 21:08:04.959285 1716 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 21:08:04.959317 1716 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 21:08:04.959354 1716 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 21:08:04.960598 1716 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38369
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790284076800039-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790284076800039-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790284076800039-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790284076800039-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:39063
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1716
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790284076800039-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 19:49:06 UTC on ccycloud.quasar-vhqjoc.root.comops.site
build id 4670864
I20260924 21:08:04.960816 1716 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 21:08:04.961033 1716 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 21:08:04.965276 1721 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 21:08:04.965415 1722 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 21:08:04.965400 1716 server_base.cc:1034] running on GCE node
W20260924 21:08:04.965337 1724 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 21:08:04.965664 1716 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 21:08:04.965895 1716 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 21:08:04.967063 1716 hybrid_clock.cc:648] HybridClock initialized: now 1790284084967052 us; error 30 us; skew 500 ppm
Sep 24 21:08:04 dist-test-slave-c7tk krb5kdc[1459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284084, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 21:08:04.969249 1716 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 21:08:04.969588 1716 webserver.cc:460] Webserver started at http://127.0.98.2:35967/ using document root <none> and password file <none>
I20260924 21:08:04.969774 1716 fs_manager.cc:359] Metadata directory not provided
I20260924 21:08:04.969856 1716 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 21:08:04.969969 1716 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 21:08:04.970764 1716 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790284076800039-392-0/minicluster-data/ts-1/data/instance:
uuid: "59e4d44abd324614878413675cd9ece6"
format_stamp: "Formatted at 2026-09-24 21:08:04 on dist-test-slave-c7tk"
server_key: "3a1f0139f130a06a9754a30ed2e23b98"
server_key_iv: "c1101bcdb36594e1bbb552547f4502b7"
server_key_version: "encryptionkey@0"
I20260924 21:08:04.971086 1716 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790284076800039-392-0/minicluster-data/ts-1/wal/instance:
uuid: "59e4d44abd324614878413675cd9ece6"
format_stamp: "Formatted at 2026-09-24 21:08:04 on dist-test-slave-c7tk"
server_key: "3a1f0139f130a06a9754a30ed2e23b98"
server_key_iv: "c1101bcdb36594e1bbb552547f4502b7"
server_key_version: "encryptionkey@0"
I20260924 21:08:04.972195 1716 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 21:08:04.972816 1731 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 21:08:04.972982 1716 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 21:08:04.973062 1716 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790284076800039-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790284076800039-392-0/minicluster-data/ts-1/wal
uuid: "59e4d44abd324614878413675cd9ece6"
format_stamp: "Formatted at 2026-09-24 21:08:04 on dist-test-slave-c7tk"
server_key: "3a1f0139f130a06a9754a30ed2e23b98"
server_key_iv: "c1101bcdb36594e1bbb552547f4502b7"
server_key_version: "encryptionkey@0"
I20260924 21:08:04.973117 1716 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790284076800039-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790284076800039-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790284076800039-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 21:08:04.986655 1716 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 21:08:04.988649 1716 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 21:08:04.988782 1716 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 21:08:04.989161 1716 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 21:08:04.989220 1716 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 21:08:04.989260 1716 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 21:08:04.989287 1716 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 21:08:04.995587 1716 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:45399
Sep 24 21:08:04 dist-test-slave-c7tk krb5kdc[1459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284084, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 21:08:04.995643 1846 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:45399 every 8 connection(s)
I20260924 21:08:04.995883 1716 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790284076800039-392-0/minicluster-data/ts-1/data/info.pb
I20260924 21:08:04.998798 1553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:04.989643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:50915 (local address 127.0.98.62:39063)
0924 21:08:04.989691 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0924 21:08:04.989692 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 21:08:04.989929 (+ 237us) server_negotiation.cc:407] Connection header received
0924 21:08:04.990405 (+ 476us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:08:04.990407 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:08:04.990421 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:08:04.990507 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:08:04.991335 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:04.992093 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:04.993489 (+ 1396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:04.993854 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:04.993946 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:04.996125 (+ 2179us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 21:08:04.996127 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 21:08:04.996127 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 21:08:04.996135 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 21:08:04.997507 (+ 1372us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:08:04.997994 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:08:04.997996 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:08:04.997996 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:08:04.998047 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:08:04.998263 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:08:04.998264 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:08:04.998265 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:08:04.998364 (+ 99us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 21:08:04.998431 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0924 21:08:04.998656 (+ 225us) server_negotiation.cc:299] Negotiation successful
0924 21:08:04.998719 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260924 21:08:04.998946 1740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:04.989620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39063 (local address 127.0.98.2:50915)
0924 21:08:04.989764 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0924 21:08:04.989841 (+ 77us) client_negotiation.cc:174] Beginning negotiation
0924 21:08:04.990284 (+ 443us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 21:08:04.990570 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 21:08:04.990574 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 21:08:04.990923 (+ 349us) client_negotiation.cc:189] Negotiated authn=SASL
0924 21:08:04.991179 (+ 256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:04.991184 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:04.992217 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:04.992218 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:04.993401 (+ 1183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:04.993403 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:04.993954 (+ 551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:04.993955 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:04.994073 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:04.994510 (+ 437us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 21:08:04.994518 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 21:08:04.996045 (+ 1527us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 21:08:04.997608 (+ 1563us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 21:08:04.997613 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 21:08:04.997615 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 21:08:04.997920 (+ 305us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 21:08:04.998110 (+ 190us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 21:08:04.998110 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 21:08:04.998111 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 21:08:04.998206 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 21:08:04.998435 (+ 229us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 21:08:04.998440 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 21:08:04.998550 (+ 110us) client_negotiation.cc:769] Sending connection context
0924 21:08:04.998638 (+ 88us) client_negotiation.cc:240] Negotiation successful
0924 21:08:04.998757 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":71,"threads_started":1}
I20260924 21:08:04.999399 1847 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39063
I20260924 21:08:04.999514 1847 heartbeater.cc:464] Registering TS with master...
I20260924 21:08:04.999670 1847 heartbeater.cc:511] Master 127.0.98.62:39063 requested a full tablet report, sending...
I20260924 21:08:05.000061 1504 ts_manager.cc:194] Registered new tserver with Master: 59e4d44abd324614878413675cd9ece6 (127.0.98.2:45399)
I20260924 21:08:05.000509 1504 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:50915
I20260924 21:08:05.001525 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAzNNjS/build/release/bin/kudu as pid 1716
I20260924 21:08:05.001613 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790284076800039-392-0/minicluster-data/ts-1/wal/instance
I20260924 21:08:05.001755 392 external_mini_cluster.cc:1428] Setting key 10352b13db1a8a40bd7e8924f8c811b2
I20260924 21:08:05.005430 1553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:05.000314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:45705 (local address 127.0.98.62:39063)
0924 21:08:05.000346 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0924 21:08:05.000347 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 21:08:05.000416 (+ 69us) server_negotiation.cc:407] Connection header received
0924 21:08:05.000474 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:08:05.000474 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:08:05.000480 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:08:05.000517 (+ 37us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:08:05.000836 (+ 319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:05.001391 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:05.002237 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:05.002458 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:05.002522 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:05.003185 (+ 663us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 21:08:05.003186 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 21:08:05.003187 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 21:08:05.003193 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 21:08:05.004376 (+ 1183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:08:05.004687 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:08:05.004687 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:08:05.004688 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:08:05.004730 (+ 42us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:08:05.004906 (+ 176us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:08:05.004907 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:08:05.004908 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:08:05.005042 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 21:08:05.005118 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0924 21:08:05.005295 (+ 177us) server_negotiation.cc:299] Negotiation successful
0924 21:08:05.005358 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260924 21:08:05.005460 1853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:05.000234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39063 (local address 127.0.98.2:45705)
0924 21:08:05.000358 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0924 21:08:05.000368 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 21:08:05.000423 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 21:08:05.000523 (+ 100us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 21:08:05.000523 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 21:08:05.000705 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0924 21:08:05.000764 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:05.000766 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:05.001489 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:05.001490 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:05.002153 (+ 663us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:05.002155 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:05.002525 (+ 370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:05.002525 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:05.002565 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:05.002901 (+ 336us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 21:08:05.002905 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 21:08:05.003114 (+ 209us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 21:08:05.004473 (+ 1359us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 21:08:05.004474 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 21:08:05.004474 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 21:08:05.004618 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 21:08:05.004792 (+ 174us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 21:08:05.004793 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 21:08:05.004793 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 21:08:05.004841 (+ 48us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 21:08:05.005161 (+ 320us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 21:08:05.005162 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 21:08:05.005204 (+ 42us) client_negotiation.cc:769] Sending connection context
0924 21:08:05.005272 (+ 68us) client_negotiation.cc:240] Negotiation successful
0924 21:08:05.005355 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":77,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790284076800039-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790284076800039-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260924 21:08:05.023783 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAzNNjS/build/release/bin/kudu
/tmp/dist-test-taskAzNNjS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790284076800039-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790284076800039-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790284076800039-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790284076800039-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39063
--keytab_file=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790284076800039-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38369
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790284076800039-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790284076800039-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790284076800039-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 21:08:05.094370 1856 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 21:08:05.094521 1856 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 21:08:05.094540 1856 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 21:08:05.095500 1856 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 21:08:05.095537 1856 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 21:08:05.095554 1856 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 21:08:05.095602 1856 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 21:08:05.096854 1856 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38369
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790284076800039-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790284076800039-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790284076800039-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790284076800039-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:39063
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1856
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790284076800039-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 19:49:06 UTC on ccycloud.quasar-vhqjoc.root.comops.site
build id 4670864
I20260924 21:08:05.097067 1856 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 21:08:05.097282 1856 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 21:08:05.101478 1862 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 21:08:05.101649 1861 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 21:08:05.101652 1856 server_base.cc:1034] running on GCE node
W20260924 21:08:05.101763 1864 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 21:08:05.101958 1856 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 21:08:05.102216 1856 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 21:08:05.103410 1856 hybrid_clock.cc:648] HybridClock initialized: now 1790284085103390 us; error 26 us; skew 500 ppm
Sep 24 21:08:05 dist-test-slave-c7tk krb5kdc[1459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284085, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 21:08:05.105501 1856 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 21:08:05.105939 1856 webserver.cc:460] Webserver started at http://127.0.98.3:40175/ using document root <none> and password file <none>
I20260924 21:08:05.106137 1856 fs_manager.cc:359] Metadata directory not provided
I20260924 21:08:05.106191 1856 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 21:08:05.106300 1856 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 21:08:05.107203 1856 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790284076800039-392-0/minicluster-data/ts-2/data/instance:
uuid: "f896991b69e44a1bbb7b271911bdb640"
format_stamp: "Formatted at 2026-09-24 21:08:05 on dist-test-slave-c7tk"
server_key: "6a9c935b66a8a90de99c9cc2d95a5518"
server_key_iv: "0efc60380d0abcc703e9a1e1bec8ea69"
server_key_version: "encryptionkey@0"
I20260924 21:08:05.107546 1856 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790284076800039-392-0/minicluster-data/ts-2/wal/instance:
uuid: "f896991b69e44a1bbb7b271911bdb640"
format_stamp: "Formatted at 2026-09-24 21:08:05 on dist-test-slave-c7tk"
server_key: "6a9c935b66a8a90de99c9cc2d95a5518"
server_key_iv: "0efc60380d0abcc703e9a1e1bec8ea69"
server_key_version: "encryptionkey@0"
I20260924 21:08:05.108697 1856 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 21:08:05.109337 1871 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 21:08:05.109498 1856 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 21:08:05.109581 1856 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790284076800039-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790284076800039-392-0/minicluster-data/ts-2/wal
uuid: "f896991b69e44a1bbb7b271911bdb640"
format_stamp: "Formatted at 2026-09-24 21:08:05 on dist-test-slave-c7tk"
server_key: "6a9c935b66a8a90de99c9cc2d95a5518"
server_key_iv: "0efc60380d0abcc703e9a1e1bec8ea69"
server_key_version: "encryptionkey@0"
I20260924 21:08:05.109656 1856 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790284076800039-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790284076800039-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790284076800039-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 21:08:05.123199 1856 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 21:08:05.124737 1856 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 21:08:05.124887 1856 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 21:08:05.125514 1856 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 21:08:05.125571 1856 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 21:08:05.125610 1856 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 21:08:05.125634 1856 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 21:08:05 dist-test-slave-c7tk krb5kdc[1459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284085, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 21:08:05.132360 1856 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:36597
I20260924 21:08:05.132447 1986 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:36597 every 8 connection(s)
I20260924 21:08:05.132716 1856 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790284076800039-392-0/minicluster-data/ts-2/data/info.pb
I20260924 21:08:05.134070 1553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:05.125879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:35829 (local address 127.0.98.62:39063)
0924 21:08:05.125942 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0924 21:08:05.125945 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 21:08:05.126167 (+ 222us) server_negotiation.cc:407] Connection header received
0924 21:08:05.126609 (+ 442us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:08:05.126610 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:08:05.126622 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:08:05.126680 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:08:05.127203 (+ 523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:05.127739 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:05.128734 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:05.128939 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:05.129023 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:05.131244 (+ 2221us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 21:08:05.131245 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 21:08:05.131246 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 21:08:05.131253 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 21:08:05.132804 (+ 1551us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:08:05.133210 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:08:05.133212 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:08:05.133212 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:08:05.133267 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:08:05.133528 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:08:05.133528 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:08:05.133528 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:08:05.133646 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 21:08:05.133697 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0924 21:08:05.133960 (+ 263us) server_negotiation.cc:299] Negotiation successful
0924 21:08:05.134014 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260924 21:08:05.134192 1880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:05.125918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39063 (local address 127.0.98.3:35829)
0924 21:08:05.126045 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0924 21:08:05.126080 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0924 21:08:05.126506 (+ 426us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 21:08:05.126741 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 21:08:05.126744 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 21:08:05.126949 (+ 205us) client_negotiation.cc:189] Negotiated authn=SASL
0924 21:08:05.127107 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:05.127110 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:05.127827 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:05.127828 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:05.128641 (+ 813us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:05.128643 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:05.129023 (+ 380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:05.129024 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:05.129125 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:05.129571 (+ 446us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 21:08:05.129578 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0924 21:08:05.131162 (+ 1584us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 21:08:05.132880 (+ 1718us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 21:08:05.132886 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 21:08:05.132888 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 21:08:05.133142 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 21:08:05.133350 (+ 208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 21:08:05.133351 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 21:08:05.133351 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 21:08:05.133464 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 21:08:05.133701 (+ 237us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 21:08:05.133704 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 21:08:05.133855 (+ 151us) client_negotiation.cc:769] Sending connection context
0924 21:08:05.133945 (+ 90us) client_negotiation.cc:240] Negotiation successful
0924 21:08:05.134043 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":91,"thread_start_us":67,"threads_started":1}
I20260924 21:08:05.134642 1987 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39063
I20260924 21:08:05.134706 1987 heartbeater.cc:464] Registering TS with master...
I20260924 21:08:05.134864 1987 heartbeater.cc:511] Master 127.0.98.62:39063 requested a full tablet report, sending...
I20260924 21:08:05.135131 1503 ts_manager.cc:194] Registered new tserver with Master: f896991b69e44a1bbb7b271911bdb640 (127.0.98.3:36597)
I20260924 21:08:05.135478 1503 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:35829
I20260924 21:08:05.137465 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAzNNjS/build/release/bin/kudu as pid 1856
I20260924 21:08:05.137566 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790284076800039-392-0/minicluster-data/ts-2/wal/instance
I20260924 21:08:05.137732 392 external_mini_cluster.cc:1428] Setting key 40b6b9714c828327c3b6b6e8f3707f32
I20260924 21:08:05.138625 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 21:08:05.140489 1993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:05.135440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39063 (local address 127.0.98.3:51165)
0924 21:08:05.135564 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0924 21:08:05.135573 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 21:08:05.135628 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 21:08:05.135725 (+ 97us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 21:08:05.135725 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 21:08:05.135872 (+ 147us) client_negotiation.cc:189] Negotiated authn=SASL
0924 21:08:05.135916 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:05.135917 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:05.136516 (+ 599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:05.136516 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:05.137102 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:05.137103 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:05.137464 (+ 361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:05.137465 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:05.137527 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:05.137993 (+ 466us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 21:08:05.137997 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 21:08:05.138290 (+ 293us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 21:08:05.139632 (+ 1342us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 21:08:05.139633 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 21:08:05.139633 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 21:08:05.139826 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 21:08:05.139985 (+ 159us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 21:08:05.139986 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 21:08:05.139987 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 21:08:05.140044 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 21:08:05.140274 (+ 230us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 21:08:05.140275 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 21:08:05.140314 (+ 39us) client_negotiation.cc:769] Sending connection context
0924 21:08:05.140376 (+ 62us) client_negotiation.cc:240] Negotiation successful
0924 21:08:05.140435 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":63,"threads_started":1}
I20260924 21:08:05.140579 1553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:05.135485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:51165 (local address 127.0.98.62:39063)
0924 21:08:05.135521 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0924 21:08:05.135522 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 21:08:05.135610 (+ 88us) server_negotiation.cc:407] Connection header received
0924 21:08:05.135678 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:08:05.135678 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:08:05.135687 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:08:05.135722 (+ 35us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:08:05.136013 (+ 291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:05.136437 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:05.137179 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:05.137380 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:05.137461 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:05.138357 (+ 896us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 21:08:05.138358 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 21:08:05.138358 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 21:08:05.138365 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 21:08:05.139566 (+ 1201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:08:05.139894 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:08:05.139894 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:08:05.139894 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:08:05.139933 (+ 39us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:08:05.140092 (+ 159us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:08:05.140093 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:08:05.140093 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:08:05.140207 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 21:08:05.140267 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0924 21:08:05.140418 (+ 151us) server_negotiation.cc:299] Negotiation successful
0924 21:08:05.140483 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
Sep 24 21:08:05 dist-test-slave-c7tk krb5kdc[1459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284085, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 24 21:08:05 dist-test-slave-c7tk krb5kdc[1459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284085, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 21:08:05.165774 1553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:05.158490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41020 (local address 127.0.98.62:39063)
0924 21:08:05.158524 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0924 21:08:05.158526 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 21:08:05.158532 (+ 6us) server_negotiation.cc:407] Connection header received
0924 21:08:05.158564 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:08:05.158566 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:08:05.158575 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:08:05.158631 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:08:05.159291 (+ 660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:05.159976 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:05.161023 (+ 1047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:05.161305 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:05.161381 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:05.163819 (+ 2438us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 21:08:05.163820 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 21:08:05.163821 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 21:08:05.163827 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 21:08:05.165015 (+ 1188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:08:05.165255 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:08:05.165256 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:08:05.165256 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:08:05.165289 (+ 33us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:08:05.165412 (+ 123us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:08:05.165412 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:08:05.165412 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:08:05.165479 (+ 67us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 21:08:05.165536 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0924 21:08:05.165680 (+ 144us) server_negotiation.cc:299] Negotiation successful
0924 21:08:05.165724 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260924 21:08:05.167305 1503 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41020:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 21:08:05.168781 1503 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 21:08:05.174376 2001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:05.170504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43729 (local address 127.0.0.1:50404)
0924 21:08:05.170749 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0924 21:08:05.170757 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0924 21:08:05.170792 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 21:08:05.171176 (+ 384us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 21:08:05.171177 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 21:08:05.171179 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 21:08:05.171222 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:05.171223 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:05.172295 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:05.172296 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:05.173285 (+ 989us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:05.173287 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:05.174134 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:05.174134 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:05.174202 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:05.174213 (+ 11us) client_negotiation.cc:769] Sending connection context
0924 21:08:05.174274 (+ 61us) client_negotiation.cc:240] Negotiation successful
0924 21:08:05.174312 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":237,"mutex_wait_us":31,"thread_start_us":42,"threads_started":1}
I20260924 21:08:05.174407 2004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:05.170590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50404 (local address 127.0.98.1:43729)
0924 21:08:05.171050 (+ 460us) server_negotiation.cc:206] Beginning negotiation
0924 21:08:05.171053 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 21:08:05.171059 (+ 6us) server_negotiation.cc:407] Connection header received
0924 21:08:05.171082 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:08:05.171083 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:08:05.171093 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:08:05.171164 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 21:08:05.171322 (+ 158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:05.172220 (+ 898us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:05.173502 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:05.174054 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:05.174123 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:05.174184 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0924 21:08:05.174286 (+ 102us) server_negotiation.cc:299] Negotiation successful
0924 21:08:05.174319 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":433,"thread_start_us":68,"threads_started":1}
I20260924 21:08:05.174515 2002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:05.170509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36597 (local address 127.0.0.1:38864)
0924 21:08:05.170819 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0924 21:08:05.170826 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0924 21:08:05.170856 (+ 30us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 21:08:05.171387 (+ 531us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 21:08:05.171388 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 21:08:05.171389 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 21:08:05.171469 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:05.171471 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:05.172272 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:05.172273 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:05.173375 (+ 1102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:05.173377 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:05.174231 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:05.174232 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:05.174281 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:05.174286 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 21:08:05.174329 (+ 43us) client_negotiation.cc:240] Negotiation successful
0924 21:08:05.174372 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":301,"mutex_wait_us":83,"thread_start_us":37,"threads_started":1}
I20260924 21:08:05.174541 2005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:05.170673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38864 (local address 127.0.98.3:36597)
0924 21:08:05.171262 (+ 589us) server_negotiation.cc:206] Beginning negotiation
0924 21:08:05.171266 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 21:08:05.171274 (+ 8us) server_negotiation.cc:407] Connection header received
0924 21:08:05.171301 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:08:05.171303 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:08:05.171332 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:08:05.171390 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 21:08:05.171537 (+ 147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:05.172197 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:05.173461 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:05.174082 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:05.174141 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:05.174199 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0924 21:08:05.174409 (+ 210us) server_negotiation.cc:299] Negotiation successful
0924 21:08:05.174442 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":561,"thread_start_us":67,"threads_started":1}
I20260924 21:08:05.175021 1641 tablet_service.cc:1467] Processing CreateTablet for tablet c2f3491c4b9f439f93513ec30a99b035 (DEFAULT_TABLE table=test-table [id=c4241f5ab1c8453e90021ea82a0e5476]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 21:08:05.175022 1921 tablet_service.cc:1467] Processing CreateTablet for tablet c2f3491c4b9f439f93513ec30a99b035 (DEFAULT_TABLE table=test-table [id=c4241f5ab1c8453e90021ea82a0e5476]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 21:08:05.175312 1641 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c2f3491c4b9f439f93513ec30a99b035. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 21:08:05.175320 1921 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c2f3491c4b9f439f93513ec30a99b035. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 21:08:05.178015 2007 tablet_bootstrap.cc:492] T c2f3491c4b9f439f93513ec30a99b035 P f896991b69e44a1bbb7b271911bdb640: Bootstrap starting.
I20260924 21:08:05.178107 2003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:05.170503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45399 (local address 127.0.0.1:40130)
0924 21:08:05.170632 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0924 21:08:05.170664 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0924 21:08:05.170733 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 21:08:05.170974 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 21:08:05.170976 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 21:08:05.170977 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 21:08:05.171013 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:05.171014 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:05.172214 (+ 1200us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:05.172215 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:05.173406 (+ 1191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:05.173407 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:05.174335 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:05.174335 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:05.174377 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:05.174379 (+ 2us) client_negotiation.cc:769] Sending connection context
0924 21:08:05.177907 (+ 3528us) client_negotiation.cc:240] Negotiation successful
0924 21:08:05.178038 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"mutex_wait_us":61,"thread_start_us":48,"threads_started":1}
I20260924 21:08:05.178228 2008 tablet_bootstrap.cc:492] T c2f3491c4b9f439f93513ec30a99b035 P c7395ba0e4ad43ea965a4a9847899e4e: Bootstrap starting.
I20260924 21:08:05.178388 2006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:05.170697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40130 (local address 127.0.98.2:45399)
0924 21:08:05.170836 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0924 21:08:05.170841 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 21:08:05.170855 (+ 14us) server_negotiation.cc:407] Connection header received
0924 21:08:05.170898 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:08:05.170899 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:08:05.170909 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:08:05.170968 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 21:08:05.171078 (+ 110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:05.172142 (+ 1064us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:05.173546 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:05.174189 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:05.174250 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:05.174307 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0924 21:08:05.178244 (+ 3937us) server_negotiation.cc:299] Negotiation successful
0924 21:08:05.178293 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116,"thread_start_us":62,"threads_started":1}
I20260924 21:08:05.178556 2007 tablet_bootstrap.cc:654] T c2f3491c4b9f439f93513ec30a99b035 P f896991b69e44a1bbb7b271911bdb640: Neither blocks nor log segments found. Creating new log.
I20260924 21:08:05.178717 2008 tablet_bootstrap.cc:654] T c2f3491c4b9f439f93513ec30a99b035 P c7395ba0e4ad43ea965a4a9847899e4e: Neither blocks nor log segments found. Creating new log.
I20260924 21:08:05.178896 2007 log.cc:824] T c2f3491c4b9f439f93513ec30a99b035 P f896991b69e44a1bbb7b271911bdb640: Log is configured to *not* fsync() on all Append() calls
I20260924 21:08:05.178870 1781 tablet_service.cc:1467] Processing CreateTablet for tablet c2f3491c4b9f439f93513ec30a99b035 (DEFAULT_TABLE table=test-table [id=c4241f5ab1c8453e90021ea82a0e5476]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 21:08:05.178962 2008 log.cc:824] T c2f3491c4b9f439f93513ec30a99b035 P c7395ba0e4ad43ea965a4a9847899e4e: Log is configured to *not* fsync() on all Append() calls
I20260924 21:08:05.179102 1781 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c2f3491c4b9f439f93513ec30a99b035. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 21:08:05.179493 2007 tablet_bootstrap.cc:492] T c2f3491c4b9f439f93513ec30a99b035 P f896991b69e44a1bbb7b271911bdb640: No bootstrap required, opened a new log
I20260924 21:08:05.179529 2007 ts_tablet_manager.cc:1397] T c2f3491c4b9f439f93513ec30a99b035 P f896991b69e44a1bbb7b271911bdb640: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 21:08:05.180568 2007 raft_consensus.cc:348] T c2f3491c4b9f439f93513ec30a99b035 P f896991b69e44a1bbb7b271911bdb640 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7395ba0e4ad43ea965a4a9847899e4e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43729 } } peers { permanent_uuid: "59e4d44abd324614878413675cd9ece6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45399 } } peers { permanent_uuid: "f896991b69e44a1bbb7b271911bdb640" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36597 } }
I20260924 21:08:05.180661 2007 raft_consensus.cc:374] T c2f3491c4b9f439f93513ec30a99b035 P f896991b69e44a1bbb7b271911bdb640 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 21:08:05.180678 2007 raft_consensus.cc:729] T c2f3491c4b9f439f93513ec30a99b035 P f896991b69e44a1bbb7b271911bdb640 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f896991b69e44a1bbb7b271911bdb640, State: Initialized, Role: FOLLOWER
I20260924 21:08:05.180749 2007 consensus_queue.cc:260] T c2f3491c4b9f439f93513ec30a99b035 P f896991b69e44a1bbb7b271911bdb640 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7395ba0e4ad43ea965a4a9847899e4e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43729 } } peers { permanent_uuid: "59e4d44abd324614878413675cd9ece6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45399 } } peers { permanent_uuid: "f896991b69e44a1bbb7b271911bdb640" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36597 } }
I20260924 21:08:05.180953 2007 ts_tablet_manager.cc:1428] T c2f3491c4b9f439f93513ec30a99b035 P f896991b69e44a1bbb7b271911bdb640: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260924 21:08:05.180996 1987 heartbeater.cc:503] Master 127.0.98.62:39063 was elected leader, sending a full tablet report...
I20260924 21:08:05.181268 2008 tablet_bootstrap.cc:492] T c2f3491c4b9f439f93513ec30a99b035 P c7395ba0e4ad43ea965a4a9847899e4e: No bootstrap required, opened a new log
I20260924 21:08:05.181314 2008 ts_tablet_manager.cc:1397] T c2f3491c4b9f439f93513ec30a99b035 P c7395ba0e4ad43ea965a4a9847899e4e: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260924 21:08:05.182709 2008 raft_consensus.cc:348] T c2f3491c4b9f439f93513ec30a99b035 P c7395ba0e4ad43ea965a4a9847899e4e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7395ba0e4ad43ea965a4a9847899e4e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43729 } } peers { permanent_uuid: "59e4d44abd324614878413675cd9ece6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45399 } } peers { permanent_uuid: "f896991b69e44a1bbb7b271911bdb640" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36597 } }
I20260924 21:08:05.182811 2008 raft_consensus.cc:374] T c2f3491c4b9f439f93513ec30a99b035 P c7395ba0e4ad43ea965a4a9847899e4e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 21:08:05.182828 2008 raft_consensus.cc:729] T c2f3491c4b9f439f93513ec30a99b035 P c7395ba0e4ad43ea965a4a9847899e4e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c7395ba0e4ad43ea965a4a9847899e4e, State: Initialized, Role: FOLLOWER
I20260924 21:08:05.182945 2008 consensus_queue.cc:260] T c2f3491c4b9f439f93513ec30a99b035 P c7395ba0e4ad43ea965a4a9847899e4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7395ba0e4ad43ea965a4a9847899e4e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43729 } } peers { permanent_uuid: "59e4d44abd324614878413675cd9ece6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45399 } } peers { permanent_uuid: "f896991b69e44a1bbb7b271911bdb640" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36597 } }
I20260924 21:08:05.183156 2008 ts_tablet_manager.cc:1428] T c2f3491c4b9f439f93513ec30a99b035 P c7395ba0e4ad43ea965a4a9847899e4e: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260924 21:08:05.183207 1707 heartbeater.cc:503] Master 127.0.98.62:39063 was elected leader, sending a full tablet report...
I20260924 21:08:05.183326 2012 tablet_bootstrap.cc:492] T c2f3491c4b9f439f93513ec30a99b035 P 59e4d44abd324614878413675cd9ece6: Bootstrap starting.
I20260924 21:08:05.183897 2012 tablet_bootstrap.cc:654] T c2f3491c4b9f439f93513ec30a99b035 P 59e4d44abd324614878413675cd9ece6: Neither blocks nor log segments found. Creating new log.
I20260924 21:08:05.184151 2012 log.cc:824] T c2f3491c4b9f439f93513ec30a99b035 P 59e4d44abd324614878413675cd9ece6: Log is configured to *not* fsync() on all Append() calls
I20260924 21:08:05.184669 2012 tablet_bootstrap.cc:492] T c2f3491c4b9f439f93513ec30a99b035 P 59e4d44abd324614878413675cd9ece6: No bootstrap required, opened a new log
I20260924 21:08:05.184711 2012 ts_tablet_manager.cc:1397] T c2f3491c4b9f439f93513ec30a99b035 P 59e4d44abd324614878413675cd9ece6: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260924 21:08:05.185786 2012 raft_consensus.cc:348] T c2f3491c4b9f439f93513ec30a99b035 P 59e4d44abd324614878413675cd9ece6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7395ba0e4ad43ea965a4a9847899e4e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43729 } } peers { permanent_uuid: "59e4d44abd324614878413675cd9ece6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45399 } } peers { permanent_uuid: "f896991b69e44a1bbb7b271911bdb640" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36597 } }
I20260924 21:08:05.185971 2012 raft_consensus.cc:374] T c2f3491c4b9f439f93513ec30a99b035 P 59e4d44abd324614878413675cd9ece6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 21:08:05.186013 2012 raft_consensus.cc:729] T c2f3491c4b9f439f93513ec30a99b035 P 59e4d44abd324614878413675cd9ece6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59e4d44abd324614878413675cd9ece6, State: Initialized, Role: FOLLOWER
I20260924 21:08:05.186141 2012 consensus_queue.cc:260] T c2f3491c4b9f439f93513ec30a99b035 P 59e4d44abd324614878413675cd9ece6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7395ba0e4ad43ea965a4a9847899e4e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43729 } } peers { permanent_uuid: "59e4d44abd324614878413675cd9ece6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45399 } } peers { permanent_uuid: "f896991b69e44a1bbb7b271911bdb640" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36597 } }
I20260924 21:08:05.186355 2012 ts_tablet_manager.cc:1428] T c2f3491c4b9f439f93513ec30a99b035 P 59e4d44abd324614878413675cd9ece6: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260924 21:08:05.186383 1847 heartbeater.cc:503] Master 127.0.98.62:39063 was elected leader, sending a full tablet report...
W20260924 21:08:05.246594 1848 tablet.cc:2369] T c2f3491c4b9f439f93513ec30a99b035 P 59e4d44abd324614878413675cd9ece6: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 21:08:05.361267 1708 tablet.cc:2369] T c2f3491c4b9f439f93513ec30a99b035 P c7395ba0e4ad43ea965a4a9847899e4e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 21:08:05.383179 1988 tablet.cc:2369] T c2f3491c4b9f439f93513ec30a99b035 P f896991b69e44a1bbb7b271911bdb640: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 21:08:05.438696 2011 raft_consensus.cc:482] T c2f3491c4b9f439f93513ec30a99b035 P f896991b69e44a1bbb7b271911bdb640 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 21:08:05.438820 2011 raft_consensus.cc:504] T c2f3491c4b9f439f93513ec30a99b035 P f896991b69e44a1bbb7b271911bdb640 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7395ba0e4ad43ea965a4a9847899e4e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43729 } } peers { permanent_uuid: "59e4d44abd324614878413675cd9ece6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45399 } } peers { permanent_uuid: "f896991b69e44a1bbb7b271911bdb640" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36597 } }
I20260924 21:08:05.439257 2011 leader_election.cc:288] T c2f3491c4b9f439f93513ec30a99b035 P f896991b69e44a1bbb7b271911bdb640 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c7395ba0e4ad43ea965a4a9847899e4e (127.0.98.1:43729), 59e4d44abd324614878413675cd9ece6 (127.0.98.2:45399)
I20260924 21:08:05.443010 2004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:05.439491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:58707 (local address 127.0.98.1:43729)
0924 21:08:05.439533 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0924 21:08:05.439535 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 21:08:05.439776 (+ 241us) server_negotiation.cc:407] Connection header received
0924 21:08:05.439800 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:08:05.439801 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:08:05.439811 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:08:05.439873 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 21:08:05.440023 (+ 150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:05.440739 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:05.442102 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:05.442646 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:05.442731 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:05.442741 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0924 21:08:05.442914 (+ 173us) server_negotiation.cc:299] Negotiation successful
0924 21:08:05.442945 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260924 21:08:05.443010 2006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:05.439509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:43565 (local address 127.0.98.2:45399)
0924 21:08:05.439553 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0924 21:08:05.439555 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 21:08:05.439566 (+ 11us) server_negotiation.cc:407] Connection header received
0924 21:08:05.439620 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:08:05.439621 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:08:05.439632 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:08:05.439696 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 21:08:05.439874 (+ 178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:05.440674 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:05.442103 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:05.442646 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:05.442731 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:05.442741 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0924 21:08:05.442906 (+ 165us) server_negotiation.cc:299] Negotiation successful
0924 21:08:05.442943 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260924 21:08:05.443025 2016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:05.439458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43729 (local address 127.0.98.3:58707)
0924 21:08:05.439668 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0924 21:08:05.439693 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0924 21:08:05.439750 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 21:08:05.439907 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 21:08:05.439908 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 21:08:05.439909 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 21:08:05.439960 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:05.439961 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:05.440818 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:05.440818 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:05.442016 (+ 1198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:05.442019 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:05.442738 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:05.442739 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:05.442823 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:05.442828 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 21:08:05.442904 (+ 76us) client_negotiation.cc:240] Negotiation successful
0924 21:08:05.442930 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":198,"mutex_wait_us":86,"thread_start_us":94,"threads_started":1}
I20260924 21:08:05.443033 1880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:05.439457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45399 (local address 127.0.98.3:43565)
0924 21:08:05.439501 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0924 21:08:05.439511 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 21:08:05.439560 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 21:08:05.439719 (+ 159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 21:08:05.439720 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 21:08:05.439722 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 21:08:05.439795 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:05.439796 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:05.440756 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:05.440756 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:05.442015 (+ 1259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:05.442019 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:05.442738 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:05.442739 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:05.442823 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:05.442828 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 21:08:05.442882 (+ 54us) client_negotiation.cc:240] Negotiation successful
0924 21:08:05.442925 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24,"mutex_wait_us":7}
I20260924 21:08:05.443392 1661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2f3491c4b9f439f93513ec30a99b035" candidate_uuid: "f896991b69e44a1bbb7b271911bdb640" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c7395ba0e4ad43ea965a4a9847899e4e" is_pre_election: true
I20260924 21:08:05.443562 1661 raft_consensus.cc:2454] T c2f3491c4b9f439f93513ec30a99b035 P c7395ba0e4ad43ea965a4a9847899e4e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f896991b69e44a1bbb7b271911bdb640 in term 0.
I20260924 21:08:05.443540 1801 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2f3491c4b9f439f93513ec30a99b035" candidate_uuid: "f896991b69e44a1bbb7b271911bdb640" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59e4d44abd324614878413675cd9ece6" is_pre_election: true
I20260924 21:08:05.443655 1801 raft_consensus.cc:2454] T c2f3491c4b9f439f93513ec30a99b035 P 59e4d44abd324614878413675cd9ece6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f896991b69e44a1bbb7b271911bdb640 in term 0.
I20260924 21:08:05.444047 1872 leader_election.cc:302] T c2f3491c4b9f439f93513ec30a99b035 P f896991b69e44a1bbb7b271911bdb640 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c7395ba0e4ad43ea965a4a9847899e4e, f896991b69e44a1bbb7b271911bdb640; no voters:
I20260924 21:08:05.444222 2011 raft_consensus.cc:2781] T c2f3491c4b9f439f93513ec30a99b035 P f896991b69e44a1bbb7b271911bdb640 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 21:08:05.444278 2011 raft_consensus.cc:482] T c2f3491c4b9f439f93513ec30a99b035 P f896991b69e44a1bbb7b271911bdb640 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 21:08:05.444304 2011 raft_consensus.cc:3037] T c2f3491c4b9f439f93513ec30a99b035 P f896991b69e44a1bbb7b271911bdb640 [term 0 FOLLOWER]: Advancing to term 1
I20260924 21:08:05.444954 2011 raft_consensus.cc:504] T c2f3491c4b9f439f93513ec30a99b035 P f896991b69e44a1bbb7b271911bdb640 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7395ba0e4ad43ea965a4a9847899e4e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43729 } } peers { permanent_uuid: "59e4d44abd324614878413675cd9ece6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45399 } } peers { permanent_uuid: "f896991b69e44a1bbb7b271911bdb640" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36597 } }
I20260924 21:08:05.445093 2011 leader_election.cc:288] T c2f3491c4b9f439f93513ec30a99b035 P f896991b69e44a1bbb7b271911bdb640 [CANDIDATE]: Term 1 election: Requested vote from peers c7395ba0e4ad43ea965a4a9847899e4e (127.0.98.1:43729), 59e4d44abd324614878413675cd9ece6 (127.0.98.2:45399)
I20260924 21:08:05.445252 1801 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2f3491c4b9f439f93513ec30a99b035" candidate_uuid: "f896991b69e44a1bbb7b271911bdb640" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59e4d44abd324614878413675cd9ece6"
I20260924 21:08:05.445251 1661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2f3491c4b9f439f93513ec30a99b035" candidate_uuid: "f896991b69e44a1bbb7b271911bdb640" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c7395ba0e4ad43ea965a4a9847899e4e"
I20260924 21:08:05.445333 1801 raft_consensus.cc:3037] T c2f3491c4b9f439f93513ec30a99b035 P 59e4d44abd324614878413675cd9ece6 [term 0 FOLLOWER]: Advancing to term 1
I20260924 21:08:05.445333 1661 raft_consensus.cc:3037] T c2f3491c4b9f439f93513ec30a99b035 P c7395ba0e4ad43ea965a4a9847899e4e [term 0 FOLLOWER]: Advancing to term 1
I20260924 21:08:05.446236 1661 raft_consensus.cc:2454] T c2f3491c4b9f439f93513ec30a99b035 P c7395ba0e4ad43ea965a4a9847899e4e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f896991b69e44a1bbb7b271911bdb640 in term 1.
I20260924 21:08:05.446233 1801 raft_consensus.cc:2454] T c2f3491c4b9f439f93513ec30a99b035 P 59e4d44abd324614878413675cd9ece6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f896991b69e44a1bbb7b271911bdb640 in term 1.
I20260924 21:08:05.446427 1872 leader_election.cc:302] T c2f3491c4b9f439f93513ec30a99b035 P f896991b69e44a1bbb7b271911bdb640 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c7395ba0e4ad43ea965a4a9847899e4e, f896991b69e44a1bbb7b271911bdb640; no voters:
I20260924 21:08:05.446523 2011 raft_consensus.cc:2781] T c2f3491c4b9f439f93513ec30a99b035 P f896991b69e44a1bbb7b271911bdb640 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 21:08:05.446677 2011 raft_consensus.cc:686] T c2f3491c4b9f439f93513ec30a99b035 P f896991b69e44a1bbb7b271911bdb640 [term 1 LEADER]: Becoming Leader. State: Replica: f896991b69e44a1bbb7b271911bdb640, State: Running, Role: LEADER
I20260924 21:08:05.446784 2011 consensus_queue.cc:237] T c2f3491c4b9f439f93513ec30a99b035 P f896991b69e44a1bbb7b271911bdb640 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7395ba0e4ad43ea965a4a9847899e4e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43729 } } peers { permanent_uuid: "59e4d44abd324614878413675cd9ece6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45399 } } peers { permanent_uuid: "f896991b69e44a1bbb7b271911bdb640" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36597 } }
I20260924 21:08:05.447587 1503 catalog_manager.cc:5495] T c2f3491c4b9f439f93513ec30a99b035 P f896991b69e44a1bbb7b271911bdb640 reported cstate change: term changed from 0 to 1, leader changed from <none> to f896991b69e44a1bbb7b271911bdb640 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "f896991b69e44a1bbb7b271911bdb640" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7395ba0e4ad43ea965a4a9847899e4e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43729 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "59e4d44abd324614878413675cd9ece6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45399 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f896991b69e44a1bbb7b271911bdb640" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36597 } health_report { overall_health: HEALTHY } } }
I20260924 21:08:05.454569 2005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:05.452471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38878 (local address 127.0.98.3:36597)
0924 21:08:05.452517 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0924 21:08:05.452519 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 21:08:05.452575 (+ 56us) server_negotiation.cc:407] Connection header received
0924 21:08:05.452641 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:08:05.452642 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:08:05.452655 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:08:05.452708 (+ 53us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 21:08:05.452827 (+ 119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:05.453236 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:05.453980 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:05.454148 (+ 168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:05.454195 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:05.454341 (+ 146us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 21:08:05.454416 (+ 75us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 21:08:05.454468 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0924 21:08:05.454508 (+ 40us) server_negotiation.cc:299] Negotiation successful
0924 21:08:05.454522 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260924 21:08:05.455940 1661 raft_consensus.cc:1261] T c2f3491c4b9f439f93513ec30a99b035 P c7395ba0e4ad43ea965a4a9847899e4e [term 1 FOLLOWER]: Refusing update from remote peer f896991b69e44a1bbb7b271911bdb640: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 21:08:05.455969 1801 raft_consensus.cc:1261] T c2f3491c4b9f439f93513ec30a99b035 P 59e4d44abd324614878413675cd9ece6 [term 1 FOLLOWER]: Refusing update from remote peer f896991b69e44a1bbb7b271911bdb640: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 21:08:05.456236 2017 consensus_queue.cc:1035] T c2f3491c4b9f439f93513ec30a99b035 P f896991b69e44a1bbb7b271911bdb640 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c7395ba0e4ad43ea965a4a9847899e4e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43729 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 21:08:05.456327 2011 consensus_queue.cc:1035] T c2f3491c4b9f439f93513ec30a99b035 P f896991b69e44a1bbb7b271911bdb640 [LEADER]: Connected to new peer: Peer: permanent_uuid: "59e4d44abd324614878413675cd9ece6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 21:08:05.458374 2020 mvcc.cc:205] Tried to move back new op lower bound from 7333003614025498624 to 7333003613990674432. Current Snapshot: MvccSnapshot[applied={T|T < 7333003614025498624}]
I20260924 21:08:05.458994 2022 mvcc.cc:205] Tried to move back new op lower bound from 7333003614025498624 to 7333003613990674432. Current Snapshot: MvccSnapshot[applied={T|T < 7333003614025498624}]
I20260924 21:08:05.461566 1503 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41020:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 21:08:05.461648 1503 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41020:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 21:08:05.462298 1503 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ecbe9754db8046bea8afe31efa3a107a: Sending DeleteTablet for 3 replicas of tablet c2f3491c4b9f439f93513ec30a99b035
I20260924 21:08:05.462642 1921 tablet_service.cc:1514] Processing DeleteTablet for tablet c2f3491c4b9f439f93513ec30a99b035 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 21:08:05 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:38864
I20260924 21:08:05.462647 1641 tablet_service.cc:1514] Processing DeleteTablet for tablet c2f3491c4b9f439f93513ec30a99b035 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 21:08:05 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:50404
I20260924 21:08:05.462724 1781 tablet_service.cc:1514] Processing DeleteTablet for tablet c2f3491c4b9f439f93513ec30a99b035 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 21:08:05 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:40130
I20260924 21:08:05.462958 2029 tablet_replica.cc:330] T c2f3491c4b9f439f93513ec30a99b035 P c7395ba0e4ad43ea965a4a9847899e4e: stopping tablet replica
I20260924 21:08:05.463080 2029 raft_consensus.cc:2229] T c2f3491c4b9f439f93513ec30a99b035 P c7395ba0e4ad43ea965a4a9847899e4e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 21:08:05.463052 1503 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41020:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260924 21:08:05.463120 2031 tablet_replica.cc:330] T c2f3491c4b9f439f93513ec30a99b035 P 59e4d44abd324614878413675cd9ece6: stopping tablet replica
W20260924 21:08:05.463182 1503 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 21:08:05.463197 2031 raft_consensus.cc:2229] T c2f3491c4b9f439f93513ec30a99b035 P 59e4d44abd324614878413675cd9ece6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 21:08:05.463199 2029 raft_consensus.cc:2258] T c2f3491c4b9f439f93513ec30a99b035 P c7395ba0e4ad43ea965a4a9847899e4e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 21:08:05.463282 2031 raft_consensus.cc:2258] T c2f3491c4b9f439f93513ec30a99b035 P 59e4d44abd324614878413675cd9ece6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 21:08:05.463614 2029 ts_tablet_manager.cc:1905] T c2f3491c4b9f439f93513ec30a99b035 P c7395ba0e4ad43ea965a4a9847899e4e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 21:08:05.463682 2031 ts_tablet_manager.cc:1905] T c2f3491c4b9f439f93513ec30a99b035 P 59e4d44abd324614878413675cd9ece6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 21:08:05.464684 1781 tablet_service.cc:1467] Processing CreateTablet for tablet a3a0a564d3a14700a68ea68427f338bf (DEFAULT_TABLE table=test-table [id=d3bd332c0ab841cf95204839b352a118]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 21:08:05.464776 1781 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a3a0a564d3a14700a68ea68427f338bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 21:08:05.465189 2029 ts_tablet_manager.cc:1918] T c2f3491c4b9f439f93513ec30a99b035 P c7395ba0e4ad43ea965a4a9847899e4e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 21:08:05.465210 2031 ts_tablet_manager.cc:1918] T c2f3491c4b9f439f93513ec30a99b035 P 59e4d44abd324614878413675cd9ece6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 21:08:05.465279 2031 log.cc:1196] T c2f3491c4b9f439f93513ec30a99b035 P 59e4d44abd324614878413675cd9ece6: Deleting WAL directory at /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790284076800039-392-0/minicluster-data/ts-1/wal/wals/c2f3491c4b9f439f93513ec30a99b035
I20260924 21:08:05.465284 2029 log.cc:1196] T c2f3491c4b9f439f93513ec30a99b035 P c7395ba0e4ad43ea965a4a9847899e4e: Deleting WAL directory at /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790284076800039-392-0/minicluster-data/ts-0/wal/wals/c2f3491c4b9f439f93513ec30a99b035
I20260924 21:08:05.465534 2031 ts_tablet_manager.cc:1939] T c2f3491c4b9f439f93513ec30a99b035 P 59e4d44abd324614878413675cd9ece6: Deleting consensus metadata
I20260924 21:08:05.465538 2029 ts_tablet_manager.cc:1939] T c2f3491c4b9f439f93513ec30a99b035 P c7395ba0e4ad43ea965a4a9847899e4e: Deleting consensus metadata
I20260924 21:08:05.465893 2012 tablet_bootstrap.cc:492] T a3a0a564d3a14700a68ea68427f338bf P 59e4d44abd324614878413675cd9ece6: Bootstrap starting.
I20260924 21:08:05.465922 1488 catalog_manager.cc:4841] TS c7395ba0e4ad43ea965a4a9847899e4e (127.0.98.1:43729): tablet c2f3491c4b9f439f93513ec30a99b035 (table test-table [id=c4241f5ab1c8453e90021ea82a0e5476]) successfully deleted
I20260924 21:08:05.466187 2012 tablet_bootstrap.cc:654] T a3a0a564d3a14700a68ea68427f338bf P 59e4d44abd324614878413675cd9ece6: Neither blocks nor log segments found. Creating new log.
I20260924 21:08:05.467053 1489 catalog_manager.cc:4841] TS 59e4d44abd324614878413675cd9ece6 (127.0.98.2:45399): tablet c2f3491c4b9f439f93513ec30a99b035 (table test-table [id=c4241f5ab1c8453e90021ea82a0e5476]) successfully deleted
I20260924 21:08:05.467190 1641 tablet_service.cc:1467] Processing CreateTablet for tablet a3a0a564d3a14700a68ea68427f338bf (DEFAULT_TABLE table=test-table [id=d3bd332c0ab841cf95204839b352a118]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 21:08:05.467299 1641 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a3a0a564d3a14700a68ea68427f338bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 21:08:05.468147 2008 tablet_bootstrap.cc:492] T a3a0a564d3a14700a68ea68427f338bf P c7395ba0e4ad43ea965a4a9847899e4e: Bootstrap starting.
I20260924 21:08:05.468261 1921 tablet_service.cc:1467] Processing CreateTablet for tablet a3a0a564d3a14700a68ea68427f338bf (DEFAULT_TABLE table=test-table [id=d3bd332c0ab841cf95204839b352a118]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 21:08:05.468364 1921 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a3a0a564d3a14700a68ea68427f338bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 21:08:05.468482 2008 tablet_bootstrap.cc:654] T a3a0a564d3a14700a68ea68427f338bf P c7395ba0e4ad43ea965a4a9847899e4e: Neither blocks nor log segments found. Creating new log.
I20260924 21:08:05.468998 2008 tablet_bootstrap.cc:492] T a3a0a564d3a14700a68ea68427f338bf P c7395ba0e4ad43ea965a4a9847899e4e: No bootstrap required, opened a new log
I20260924 21:08:05.469048 2008 ts_tablet_manager.cc:1397] T a3a0a564d3a14700a68ea68427f338bf P c7395ba0e4ad43ea965a4a9847899e4e: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260924 21:08:05.469267 2008 raft_consensus.cc:348] T a3a0a564d3a14700a68ea68427f338bf P c7395ba0e4ad43ea965a4a9847899e4e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7395ba0e4ad43ea965a4a9847899e4e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43729 } } peers { permanent_uuid: "59e4d44abd324614878413675cd9ece6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45399 } } peers { permanent_uuid: "f896991b69e44a1bbb7b271911bdb640" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36597 } }
I20260924 21:08:05.469331 2008 raft_consensus.cc:374] T a3a0a564d3a14700a68ea68427f338bf P c7395ba0e4ad43ea965a4a9847899e4e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 21:08:05.469347 2008 raft_consensus.cc:729] T a3a0a564d3a14700a68ea68427f338bf P c7395ba0e4ad43ea965a4a9847899e4e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c7395ba0e4ad43ea965a4a9847899e4e, State: Initialized, Role: FOLLOWER
I20260924 21:08:05.469391 2008 consensus_queue.cc:260] T a3a0a564d3a14700a68ea68427f338bf P c7395ba0e4ad43ea965a4a9847899e4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7395ba0e4ad43ea965a4a9847899e4e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43729 } } peers { permanent_uuid: "59e4d44abd324614878413675cd9ece6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45399 } } peers { permanent_uuid: "f896991b69e44a1bbb7b271911bdb640" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36597 } }
I20260924 21:08:05.469480 2008 ts_tablet_manager.cc:1428] T a3a0a564d3a14700a68ea68427f338bf P c7395ba0e4ad43ea965a4a9847899e4e: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260924 21:08:05.469534 2007 tablet_bootstrap.cc:492] T a3a0a564d3a14700a68ea68427f338bf P f896991b69e44a1bbb7b271911bdb640: Bootstrap starting.
I20260924 21:08:05.469905 2007 tablet_bootstrap.cc:654] T a3a0a564d3a14700a68ea68427f338bf P f896991b69e44a1bbb7b271911bdb640: Neither blocks nor log segments found. Creating new log.
I20260924 21:08:05.470036 2012 tablet_bootstrap.cc:492] T a3a0a564d3a14700a68ea68427f338bf P 59e4d44abd324614878413675cd9ece6: No bootstrap required, opened a new log
I20260924 21:08:05.470062 2012 ts_tablet_manager.cc:1397] T a3a0a564d3a14700a68ea68427f338bf P 59e4d44abd324614878413675cd9ece6: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.001s
I20260924 21:08:05.470182 2012 raft_consensus.cc:348] T a3a0a564d3a14700a68ea68427f338bf P 59e4d44abd324614878413675cd9ece6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7395ba0e4ad43ea965a4a9847899e4e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43729 } } peers { permanent_uuid: "59e4d44abd324614878413675cd9ece6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45399 } } peers { permanent_uuid: "f896991b69e44a1bbb7b271911bdb640" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36597 } }
I20260924 21:08:05.470247 2012 raft_consensus.cc:374] T a3a0a564d3a14700a68ea68427f338bf P 59e4d44abd324614878413675cd9ece6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 21:08:05.470273 2012 raft_consensus.cc:729] T a3a0a564d3a14700a68ea68427f338bf P 59e4d44abd324614878413675cd9ece6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59e4d44abd324614878413675cd9ece6, State: Initialized, Role: FOLLOWER
I20260924 21:08:05.470320 2012 consensus_queue.cc:260] T a3a0a564d3a14700a68ea68427f338bf P 59e4d44abd324614878413675cd9ece6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7395ba0e4ad43ea965a4a9847899e4e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43729 } } peers { permanent_uuid: "59e4d44abd324614878413675cd9ece6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45399 } } peers { permanent_uuid: "f896991b69e44a1bbb7b271911bdb640" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36597 } }
I20260924 21:08:05.470407 2012 ts_tablet_manager.cc:1428] T a3a0a564d3a14700a68ea68427f338bf P 59e4d44abd324614878413675cd9ece6: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 21:08:05.470492 2007 tablet_bootstrap.cc:492] T a3a0a564d3a14700a68ea68427f338bf P f896991b69e44a1bbb7b271911bdb640: No bootstrap required, opened a new log
I20260924 21:08:05.470527 2007 ts_tablet_manager.cc:1397] T a3a0a564d3a14700a68ea68427f338bf P f896991b69e44a1bbb7b271911bdb640: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260924 21:08:05.470638 2007 raft_consensus.cc:348] T a3a0a564d3a14700a68ea68427f338bf P f896991b69e44a1bbb7b271911bdb640 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7395ba0e4ad43ea965a4a9847899e4e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43729 } } peers { permanent_uuid: "59e4d44abd324614878413675cd9ece6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45399 } } peers { permanent_uuid: "f896991b69e44a1bbb7b271911bdb640" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36597 } }
I20260924 21:08:05.470682 2007 raft_consensus.cc:374] T a3a0a564d3a14700a68ea68427f338bf P f896991b69e44a1bbb7b271911bdb640 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 21:08:05.470700 2007 raft_consensus.cc:729] T a3a0a564d3a14700a68ea68427f338bf P f896991b69e44a1bbb7b271911bdb640 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f896991b69e44a1bbb7b271911bdb640, State: Initialized, Role: FOLLOWER
I20260924 21:08:05.470738 2007 consensus_queue.cc:260] T a3a0a564d3a14700a68ea68427f338bf P f896991b69e44a1bbb7b271911bdb640 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7395ba0e4ad43ea965a4a9847899e4e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43729 } } peers { permanent_uuid: "59e4d44abd324614878413675cd9ece6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45399 } } peers { permanent_uuid: "f896991b69e44a1bbb7b271911bdb640" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36597 } }
I20260924 21:08:05.470803 2007 ts_tablet_manager.cc:1428] T a3a0a564d3a14700a68ea68427f338bf P f896991b69e44a1bbb7b271911bdb640: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 21:08:05.473891 2030 tablet_replica.cc:330] T c2f3491c4b9f439f93513ec30a99b035 P f896991b69e44a1bbb7b271911bdb640: stopping tablet replica
I20260924 21:08:05.473971 2030 raft_consensus.cc:2229] T c2f3491c4b9f439f93513ec30a99b035 P f896991b69e44a1bbb7b271911bdb640 [term 1 LEADER]: Raft consensus shutting down.
I20260924 21:08:05.474069 2030 raft_consensus.cc:2258] T c2f3491c4b9f439f93513ec30a99b035 P f896991b69e44a1bbb7b271911bdb640 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 21:08:05.474336 2030 ts_tablet_manager.cc:1905] T c2f3491c4b9f439f93513ec30a99b035 P f896991b69e44a1bbb7b271911bdb640: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 21:08:05.475412 2030 ts_tablet_manager.cc:1918] T c2f3491c4b9f439f93513ec30a99b035 P f896991b69e44a1bbb7b271911bdb640: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 21:08:05.475469 2030 log.cc:1196] T c2f3491c4b9f439f93513ec30a99b035 P f896991b69e44a1bbb7b271911bdb640: Deleting WAL directory at /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790284076800039-392-0/minicluster-data/ts-2/wal/wals/c2f3491c4b9f439f93513ec30a99b035
I20260924 21:08:05.475700 2030 ts_tablet_manager.cc:1939] T c2f3491c4b9f439f93513ec30a99b035 P f896991b69e44a1bbb7b271911bdb640: Deleting consensus metadata
I20260924 21:08:05.476001 1491 catalog_manager.cc:4841] TS f896991b69e44a1bbb7b271911bdb640 (127.0.98.3:36597): tablet c2f3491c4b9f439f93513ec30a99b035 (table test-table [id=c4241f5ab1c8453e90021ea82a0e5476]) successfully deleted
I20260924 21:08:05.509121 2015 raft_consensus.cc:482] T a3a0a564d3a14700a68ea68427f338bf P 59e4d44abd324614878413675cd9ece6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 21:08:05.509186 2015 raft_consensus.cc:504] T a3a0a564d3a14700a68ea68427f338bf P 59e4d44abd324614878413675cd9ece6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7395ba0e4ad43ea965a4a9847899e4e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43729 } } peers { permanent_uuid: "59e4d44abd324614878413675cd9ece6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45399 } } peers { permanent_uuid: "f896991b69e44a1bbb7b271911bdb640" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36597 } }
I20260924 21:08:05.509425 2015 leader_election.cc:288] T a3a0a564d3a14700a68ea68427f338bf P 59e4d44abd324614878413675cd9ece6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c7395ba0e4ad43ea965a4a9847899e4e (127.0.98.1:43729), f896991b69e44a1bbb7b271911bdb640 (127.0.98.3:36597)
I20260924 21:08:05.512424 2036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:05.509560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43729 (local address 127.0.98.2:34851)
0924 21:08:05.509661 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0924 21:08:05.509670 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 21:08:05.509722 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 21:08:05.509860 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 21:08:05.509861 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 21:08:05.509862 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 21:08:05.509925 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:05.509927 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:05.510873 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:05.510874 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:05.511680 (+ 806us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:05.511682 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:05.512200 (+ 518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:05.512201 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:05.512260 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:05.512264 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 21:08:05.512317 (+ 53us) client_negotiation.cc:240] Negotiation successful
0924 21:08:05.512353 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":87,"mutex_wait_us":38,"thread_start_us":51,"threads_started":1}
I20260924 21:08:05.512427 2004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:05.509688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:34851 (local address 127.0.98.1:43729)
0924 21:08:05.509746 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0924 21:08:05.509748 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 21:08:05.509754 (+ 6us) server_negotiation.cc:407] Connection header received
0924 21:08:05.509774 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:08:05.509775 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:08:05.509782 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:08:05.509854 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 21:08:05.509976 (+ 122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:05.510792 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:05.511759 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:05.512122 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:05.512177 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:05.512186 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0924 21:08:05.512329 (+ 143us) server_negotiation.cc:299] Negotiation successful
0924 21:08:05.512373 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260924 21:08:05.512732 1661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3a0a564d3a14700a68ea68427f338bf" candidate_uuid: "59e4d44abd324614878413675cd9ece6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c7395ba0e4ad43ea965a4a9847899e4e" is_pre_election: true
I20260924 21:08:05.512804 1661 raft_consensus.cc:2454] T a3a0a564d3a14700a68ea68427f338bf P c7395ba0e4ad43ea965a4a9847899e4e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59e4d44abd324614878413675cd9ece6 in term 0.
I20260924 21:08:05.512869 2035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:05.509560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36597 (local address 127.0.98.2:52479)
0924 21:08:05.509699 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0924 21:08:05.509707 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0924 21:08:05.509773 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 21:08:05.509923 (+ 150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 21:08:05.509924 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 21:08:05.509926 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 21:08:05.509977 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:05.509979 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:05.510853 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:05.510854 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:05.511962 (+ 1108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:05.511965 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:05.512637 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:05.512638 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:05.512712 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:05.512715 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 21:08:05.512765 (+ 50us) client_negotiation.cc:240] Negotiation successful
0924 21:08:05.512798 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"mutex_wait_us":40,"spinlock_wait_cycles":768,"thread_start_us":54,"threads_started":1}
I20260924 21:08:05.512876 2005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:05.509655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:52479 (local address 127.0.98.3:36597)
0924 21:08:05.509721 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0924 21:08:05.509723 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 21:08:05.509751 (+ 28us) server_negotiation.cc:407] Connection header received
0924 21:08:05.509840 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:08:05.509841 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:08:05.509848 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:08:05.509886 (+ 38us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 21:08:05.510075 (+ 189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:05.510763 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:05.512053 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:05.512500 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:05.512584 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:05.512606 (+ 22us) server_negotiation.cc:1035] Waiting for connection context
0924 21:08:05.512783 (+ 177us) server_negotiation.cc:299] Negotiation successful
0924 21:08:05.512830 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260924 21:08:05.512938 1732 leader_election.cc:302] T a3a0a564d3a14700a68ea68427f338bf P 59e4d44abd324614878413675cd9ece6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59e4d44abd324614878413675cd9ece6, c7395ba0e4ad43ea965a4a9847899e4e; no voters:
I20260924 21:08:05.513037 2015 raft_consensus.cc:2781] T a3a0a564d3a14700a68ea68427f338bf P 59e4d44abd324614878413675cd9ece6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 21:08:05.513067 2015 raft_consensus.cc:482] T a3a0a564d3a14700a68ea68427f338bf P 59e4d44abd324614878413675cd9ece6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 21:08:05.513089 2015 raft_consensus.cc:3037] T a3a0a564d3a14700a68ea68427f338bf P 59e4d44abd324614878413675cd9ece6 [term 0 FOLLOWER]: Advancing to term 1
I20260924 21:08:05.513125 1941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3a0a564d3a14700a68ea68427f338bf" candidate_uuid: "59e4d44abd324614878413675cd9ece6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f896991b69e44a1bbb7b271911bdb640" is_pre_election: true
I20260924 21:08:05.513214 1941 raft_consensus.cc:2454] T a3a0a564d3a14700a68ea68427f338bf P f896991b69e44a1bbb7b271911bdb640 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59e4d44abd324614878413675cd9ece6 in term 0.
I20260924 21:08:05.513705 2015 raft_consensus.cc:504] T a3a0a564d3a14700a68ea68427f338bf P 59e4d44abd324614878413675cd9ece6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7395ba0e4ad43ea965a4a9847899e4e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43729 } } peers { permanent_uuid: "59e4d44abd324614878413675cd9ece6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45399 } } peers { permanent_uuid: "f896991b69e44a1bbb7b271911bdb640" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36597 } }
I20260924 21:08:05.513854 2015 leader_election.cc:288] T a3a0a564d3a14700a68ea68427f338bf P 59e4d44abd324614878413675cd9ece6 [CANDIDATE]: Term 1 election: Requested vote from peers c7395ba0e4ad43ea965a4a9847899e4e (127.0.98.1:43729), f896991b69e44a1bbb7b271911bdb640 (127.0.98.3:36597)
I20260924 21:08:05.514011 1661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3a0a564d3a14700a68ea68427f338bf" candidate_uuid: "59e4d44abd324614878413675cd9ece6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c7395ba0e4ad43ea965a4a9847899e4e"
I20260924 21:08:05.514079 1661 raft_consensus.cc:3037] T a3a0a564d3a14700a68ea68427f338bf P c7395ba0e4ad43ea965a4a9847899e4e [term 0 FOLLOWER]: Advancing to term 1
I20260924 21:08:05.514712 1661 raft_consensus.cc:2454] T a3a0a564d3a14700a68ea68427f338bf P c7395ba0e4ad43ea965a4a9847899e4e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59e4d44abd324614878413675cd9ece6 in term 1.
I20260924 21:08:05.514811 1941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3a0a564d3a14700a68ea68427f338bf" candidate_uuid: "59e4d44abd324614878413675cd9ece6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f896991b69e44a1bbb7b271911bdb640"
I20260924 21:08:05.514880 1941 raft_consensus.cc:3037] T a3a0a564d3a14700a68ea68427f338bf P f896991b69e44a1bbb7b271911bdb640 [term 0 FOLLOWER]: Advancing to term 1
I20260924 21:08:05.514883 1732 leader_election.cc:302] T a3a0a564d3a14700a68ea68427f338bf P 59e4d44abd324614878413675cd9ece6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59e4d44abd324614878413675cd9ece6, c7395ba0e4ad43ea965a4a9847899e4e; no voters:
I20260924 21:08:05.514964 2015 raft_consensus.cc:2781] T a3a0a564d3a14700a68ea68427f338bf P 59e4d44abd324614878413675cd9ece6 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 21:08:05.515085 2015 raft_consensus.cc:686] T a3a0a564d3a14700a68ea68427f338bf P 59e4d44abd324614878413675cd9ece6 [term 1 LEADER]: Becoming Leader. State: Replica: 59e4d44abd324614878413675cd9ece6, State: Running, Role: LEADER
I20260924 21:08:05.515146 2015 consensus_queue.cc:237] T a3a0a564d3a14700a68ea68427f338bf P 59e4d44abd324614878413675cd9ece6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7395ba0e4ad43ea965a4a9847899e4e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43729 } } peers { permanent_uuid: "59e4d44abd324614878413675cd9ece6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45399 } } peers { permanent_uuid: "f896991b69e44a1bbb7b271911bdb640" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36597 } }
I20260924 21:08:05.515461 1941 raft_consensus.cc:2454] T a3a0a564d3a14700a68ea68427f338bf P f896991b69e44a1bbb7b271911bdb640 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59e4d44abd324614878413675cd9ece6 in term 1.
I20260924 21:08:05.515810 1504 catalog_manager.cc:5495] T a3a0a564d3a14700a68ea68427f338bf P 59e4d44abd324614878413675cd9ece6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 59e4d44abd324614878413675cd9ece6 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "59e4d44abd324614878413675cd9ece6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7395ba0e4ad43ea965a4a9847899e4e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43729 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "59e4d44abd324614878413675cd9ece6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45399 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f896991b69e44a1bbb7b271911bdb640" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36597 } health_report { overall_health: UNKNOWN } } }
Sep 24 21:08:05 dist-test-slave-c7tk krb5kdc[1459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284084, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 21:08:05.549681 1553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:05.542446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41034 (local address 127.0.98.62:39063)
0924 21:08:05.542490 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0924 21:08:05.542492 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 21:08:05.542597 (+ 105us) server_negotiation.cc:407] Connection header received
0924 21:08:05.542625 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:08:05.542626 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:08:05.542635 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:08:05.542695 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:08:05.543150 (+ 455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:05.543817 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:05.544478 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:05.544647 (+ 169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:05.544716 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:05.547364 (+ 2648us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 21:08:05.547365 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 21:08:05.547366 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 21:08:05.547372 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 21:08:05.548614 (+ 1242us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:08:05.549007 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:08:05.549008 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:08:05.549008 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:08:05.549068 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:08:05.549258 (+ 190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:08:05.549259 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:08:05.549259 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:08:05.549356 (+ 97us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 21:08:05.549401 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0924 21:08:05.549561 (+ 160us) server_negotiation.cc:299] Negotiation successful
0924 21:08:05.549625 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":23}
I20260924 21:08:05.549790 2044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:05.542358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39063 (local address 127.0.0.1:41034)
0924 21:08:05.542533 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0924 21:08:05.542543 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 21:08:05.542592 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 21:08:05.542705 (+ 113us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 21:08:05.542705 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 21:08:05.542954 (+ 249us) client_negotiation.cc:189] Negotiated authn=SASL
0924 21:08:05.543024 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:05.543026 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:05.543894 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:05.543895 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:05.544400 (+ 505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:05.544402 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:05.544717 (+ 315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:05.544718 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:05.544776 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:05.545586 (+ 810us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 21:08:05.545590 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 21:08:05.547269 (+ 1679us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 21:08:05.548690 (+ 1421us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 21:08:05.548696 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 21:08:05.548698 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 21:08:05.548937 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 21:08:05.549129 (+ 192us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 21:08:05.549130 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 21:08:05.549130 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 21:08:05.549199 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 21:08:05.549416 (+ 217us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 21:08:05.549421 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 21:08:05.549467 (+ 46us) client_negotiation.cc:769] Sending connection context
0924 21:08:05.549540 (+ 73us) client_negotiation.cc:240] Negotiation successful
0924 21:08:05.549702 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"mutex_wait_us":61,"thread_start_us":109,"threads_started":1}
I20260924 21:08:05.551733 1504 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:41034:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260924 21:08:05.552006 1504 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 21:08:05.553288 1921 tablet_service.cc:1467] Processing CreateTablet for tablet 94695eeed80d43e1b06988cf07fa27ff (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=d3a3dc68050e43869c8a507db308e18f]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 21:08:05.553354 1781 tablet_service.cc:1467] Processing CreateTablet for tablet 94695eeed80d43e1b06988cf07fa27ff (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=d3a3dc68050e43869c8a507db308e18f]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 21:08:05.553426 1921 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 94695eeed80d43e1b06988cf07fa27ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 21:08:05.553503 1781 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 94695eeed80d43e1b06988cf07fa27ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 21:08:05.554543 2012 tablet_bootstrap.cc:492] T 94695eeed80d43e1b06988cf07fa27ff P 59e4d44abd324614878413675cd9ece6: Bootstrap starting.
I20260924 21:08:05.554646 1641 tablet_service.cc:1467] Processing CreateTablet for tablet 94695eeed80d43e1b06988cf07fa27ff (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=d3a3dc68050e43869c8a507db308e18f]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 21:08:05.554769 1641 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 94695eeed80d43e1b06988cf07fa27ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 21:08:05.554759 2007 tablet_bootstrap.cc:492] T 94695eeed80d43e1b06988cf07fa27ff P f896991b69e44a1bbb7b271911bdb640: Bootstrap starting.
I20260924 21:08:05.554944 2012 tablet_bootstrap.cc:654] T 94695eeed80d43e1b06988cf07fa27ff P 59e4d44abd324614878413675cd9ece6: Neither blocks nor log segments found. Creating new log.
I20260924 21:08:05.555136 2007 tablet_bootstrap.cc:654] T 94695eeed80d43e1b06988cf07fa27ff P f896991b69e44a1bbb7b271911bdb640: Neither blocks nor log segments found. Creating new log.
I20260924 21:08:05.555686 2012 tablet_bootstrap.cc:492] T 94695eeed80d43e1b06988cf07fa27ff P 59e4d44abd324614878413675cd9ece6: No bootstrap required, opened a new log
I20260924 21:08:05.555734 2012 ts_tablet_manager.cc:1397] T 94695eeed80d43e1b06988cf07fa27ff P 59e4d44abd324614878413675cd9ece6: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260924 21:08:05.555852 2012 raft_consensus.cc:348] T 94695eeed80d43e1b06988cf07fa27ff P 59e4d44abd324614878413675cd9ece6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7395ba0e4ad43ea965a4a9847899e4e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43729 } } peers { permanent_uuid: "59e4d44abd324614878413675cd9ece6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45399 } } peers { permanent_uuid: "f896991b69e44a1bbb7b271911bdb640" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36597 } }
I20260924 21:08:05.555925 2012 raft_consensus.cc:374] T 94695eeed80d43e1b06988cf07fa27ff P 59e4d44abd324614878413675cd9ece6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 21:08:05.555943 2012 raft_consensus.cc:729] T 94695eeed80d43e1b06988cf07fa27ff P 59e4d44abd324614878413675cd9ece6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59e4d44abd324614878413675cd9ece6, State: Initialized, Role: FOLLOWER
I20260924 21:08:05.555972 2012 consensus_queue.cc:260] T 94695eeed80d43e1b06988cf07fa27ff P 59e4d44abd324614878413675cd9ece6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7395ba0e4ad43ea965a4a9847899e4e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43729 } } peers { permanent_uuid: "59e4d44abd324614878413675cd9ece6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45399 } } peers { permanent_uuid: "f896991b69e44a1bbb7b271911bdb640" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36597 } }
I20260924 21:08:05.556048 2012 ts_tablet_manager.cc:1428] T 94695eeed80d43e1b06988cf07fa27ff P 59e4d44abd324614878413675cd9ece6: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 21:08:05.556109 2008 tablet_bootstrap.cc:492] T 94695eeed80d43e1b06988cf07fa27ff P c7395ba0e4ad43ea965a4a9847899e4e: Bootstrap starting.
I20260924 21:08:05.556139 2015 tablet_replica.cc:439] T 94695eeed80d43e1b06988cf07fa27ff P 59e4d44abd324614878413675cd9ece6: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7395ba0e4ad43ea965a4a9847899e4e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43729 } } peers { permanent_uuid: "59e4d44abd324614878413675cd9ece6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45399 } } peers { permanent_uuid: "f896991b69e44a1bbb7b271911bdb640" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36597 } } }
I20260924 21:08:05.556200 2015 tablet_replica.cc:442] T 94695eeed80d43e1b06988cf07fa27ff P 59e4d44abd324614878413675cd9ece6: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 21:08:05.556443 2008 tablet_bootstrap.cc:654] T 94695eeed80d43e1b06988cf07fa27ff P c7395ba0e4ad43ea965a4a9847899e4e: Neither blocks nor log segments found. Creating new log.
I20260924 21:08:05.556535 2007 tablet_bootstrap.cc:492] T 94695eeed80d43e1b06988cf07fa27ff P f896991b69e44a1bbb7b271911bdb640: No bootstrap required, opened a new log
I20260924 21:08:05.556563 2007 ts_tablet_manager.cc:1397] T 94695eeed80d43e1b06988cf07fa27ff P f896991b69e44a1bbb7b271911bdb640: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260924 21:08:05.556725 2007 raft_consensus.cc:348] T 94695eeed80d43e1b06988cf07fa27ff P f896991b69e44a1bbb7b271911bdb640 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7395ba0e4ad43ea965a4a9847899e4e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43729 } } peers { permanent_uuid: "59e4d44abd324614878413675cd9ece6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45399 } } peers { permanent_uuid: "f896991b69e44a1bbb7b271911bdb640" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36597 } }
I20260924 21:08:05.556810 2007 raft_consensus.cc:374] T 94695eeed80d43e1b06988cf07fa27ff P f896991b69e44a1bbb7b271911bdb640 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 21:08:05.556838 2007 raft_consensus.cc:729] T 94695eeed80d43e1b06988cf07fa27ff P f896991b69e44a1bbb7b271911bdb640 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f896991b69e44a1bbb7b271911bdb640, State: Initialized, Role: FOLLOWER
I20260924 21:08:05.556905 2007 consensus_queue.cc:260] T 94695eeed80d43e1b06988cf07fa27ff P f896991b69e44a1bbb7b271911bdb640 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7395ba0e4ad43ea965a4a9847899e4e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43729 } } peers { permanent_uuid: "59e4d44abd324614878413675cd9ece6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45399 } } peers { permanent_uuid: "f896991b69e44a1bbb7b271911bdb640" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36597 } }
I20260924 21:08:05.556998 2007 ts_tablet_manager.cc:1428] T 94695eeed80d43e1b06988cf07fa27ff P f896991b69e44a1bbb7b271911bdb640: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260924 21:08:05.557071 2008 tablet_bootstrap.cc:492] T 94695eeed80d43e1b06988cf07fa27ff P c7395ba0e4ad43ea965a4a9847899e4e: No bootstrap required, opened a new log
I20260924 21:08:05.557071 2017 tablet_replica.cc:439] T 94695eeed80d43e1b06988cf07fa27ff P f896991b69e44a1bbb7b271911bdb640: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7395ba0e4ad43ea965a4a9847899e4e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43729 } } peers { permanent_uuid: "59e4d44abd324614878413675cd9ece6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45399 } } peers { permanent_uuid: "f896991b69e44a1bbb7b271911bdb640" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36597 } } }
I20260924 21:08:05.557123 2008 ts_tablet_manager.cc:1397] T 94695eeed80d43e1b06988cf07fa27ff P c7395ba0e4ad43ea965a4a9847899e4e: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260924 21:08:05.557128 2017 tablet_replica.cc:442] T 94695eeed80d43e1b06988cf07fa27ff P f896991b69e44a1bbb7b271911bdb640: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 21:08:05.557238 2008 raft_consensus.cc:348] T 94695eeed80d43e1b06988cf07fa27ff P c7395ba0e4ad43ea965a4a9847899e4e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7395ba0e4ad43ea965a4a9847899e4e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43729 } } peers { permanent_uuid: "59e4d44abd324614878413675cd9ece6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45399 } } peers { permanent_uuid: "f896991b69e44a1bbb7b271911bdb640" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36597 } }
I20260924 21:08:05.557315 2008 raft_consensus.cc:374] T 94695eeed80d43e1b06988cf07fa27ff P c7395ba0e4ad43ea965a4a9847899e4e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 21:08:05.557333 2008 raft_consensus.cc:729] T 94695eeed80d43e1b06988cf07fa27ff P c7395ba0e4ad43ea965a4a9847899e4e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c7395ba0e4ad43ea965a4a9847899e4e, State: Initialized, Role: FOLLOWER
I20260924 21:08:05.557372 2008 consensus_queue.cc:260] T 94695eeed80d43e1b06988cf07fa27ff P c7395ba0e4ad43ea965a4a9847899e4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7395ba0e4ad43ea965a4a9847899e4e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43729 } } peers { permanent_uuid: "59e4d44abd324614878413675cd9ece6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45399 } } peers { permanent_uuid: "f896991b69e44a1bbb7b271911bdb640" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36597 } }
I20260924 21:08:05.557456 2008 ts_tablet_manager.cc:1428] T 94695eeed80d43e1b06988cf07fa27ff P c7395ba0e4ad43ea965a4a9847899e4e: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 21:08:05.557528 2013 tablet_replica.cc:439] T 94695eeed80d43e1b06988cf07fa27ff P c7395ba0e4ad43ea965a4a9847899e4e: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7395ba0e4ad43ea965a4a9847899e4e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43729 } } peers { permanent_uuid: "59e4d44abd324614878413675cd9ece6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45399 } } peers { permanent_uuid: "f896991b69e44a1bbb7b271911bdb640" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36597 } } }
I20260924 21:08:05.557649 2013 tablet_replica.cc:442] T 94695eeed80d43e1b06988cf07fa27ff P c7395ba0e4ad43ea965a4a9847899e4e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 21:08:05.628356 2017 raft_consensus.cc:482] T 94695eeed80d43e1b06988cf07fa27ff P f896991b69e44a1bbb7b271911bdb640 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 21:08:05.628453 2017 raft_consensus.cc:504] T 94695eeed80d43e1b06988cf07fa27ff P f896991b69e44a1bbb7b271911bdb640 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7395ba0e4ad43ea965a4a9847899e4e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43729 } } peers { permanent_uuid: "59e4d44abd324614878413675cd9ece6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45399 } } peers { permanent_uuid: "f896991b69e44a1bbb7b271911bdb640" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36597 } }
I20260924 21:08:05.628654 2017 leader_election.cc:288] T 94695eeed80d43e1b06988cf07fa27ff P f896991b69e44a1bbb7b271911bdb640 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c7395ba0e4ad43ea965a4a9847899e4e (127.0.98.1:43729), 59e4d44abd324614878413675cd9ece6 (127.0.98.2:45399)
I20260924 21:08:05.628888 1661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94695eeed80d43e1b06988cf07fa27ff" candidate_uuid: "f896991b69e44a1bbb7b271911bdb640" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c7395ba0e4ad43ea965a4a9847899e4e" is_pre_election: true
I20260924 21:08:05.628907 1801 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94695eeed80d43e1b06988cf07fa27ff" candidate_uuid: "f896991b69e44a1bbb7b271911bdb640" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59e4d44abd324614878413675cd9ece6" is_pre_election: true
I20260924 21:08:05.628983 1661 raft_consensus.cc:2454] T 94695eeed80d43e1b06988cf07fa27ff P c7395ba0e4ad43ea965a4a9847899e4e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f896991b69e44a1bbb7b271911bdb640 in term 0.
I20260924 21:08:05.628983 1801 raft_consensus.cc:2454] T 94695eeed80d43e1b06988cf07fa27ff P 59e4d44abd324614878413675cd9ece6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f896991b69e44a1bbb7b271911bdb640 in term 0.
I20260924 21:08:05.629132 1872 leader_election.cc:302] T 94695eeed80d43e1b06988cf07fa27ff P f896991b69e44a1bbb7b271911bdb640 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c7395ba0e4ad43ea965a4a9847899e4e, f896991b69e44a1bbb7b271911bdb640; no voters:
I20260924 21:08:05.629282 2017 raft_consensus.cc:2781] T 94695eeed80d43e1b06988cf07fa27ff P f896991b69e44a1bbb7b271911bdb640 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 21:08:05.629328 2017 raft_consensus.cc:482] T 94695eeed80d43e1b06988cf07fa27ff P f896991b69e44a1bbb7b271911bdb640 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 21:08:05.629343 2017 raft_consensus.cc:3037] T 94695eeed80d43e1b06988cf07fa27ff P f896991b69e44a1bbb7b271911bdb640 [term 0 FOLLOWER]: Advancing to term 1
I20260924 21:08:05.630059 2017 raft_consensus.cc:504] T 94695eeed80d43e1b06988cf07fa27ff P f896991b69e44a1bbb7b271911bdb640 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7395ba0e4ad43ea965a4a9847899e4e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43729 } } peers { permanent_uuid: "59e4d44abd324614878413675cd9ece6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45399 } } peers { permanent_uuid: "f896991b69e44a1bbb7b271911bdb640" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36597 } }
I20260924 21:08:05.630223 2017 leader_election.cc:288] T 94695eeed80d43e1b06988cf07fa27ff P f896991b69e44a1bbb7b271911bdb640 [CANDIDATE]: Term 1 election: Requested vote from peers c7395ba0e4ad43ea965a4a9847899e4e (127.0.98.1:43729), 59e4d44abd324614878413675cd9ece6 (127.0.98.2:45399)
I20260924 21:08:05.630353 1661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94695eeed80d43e1b06988cf07fa27ff" candidate_uuid: "f896991b69e44a1bbb7b271911bdb640" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c7395ba0e4ad43ea965a4a9847899e4e"
I20260924 21:08:05.630367 1801 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94695eeed80d43e1b06988cf07fa27ff" candidate_uuid: "f896991b69e44a1bbb7b271911bdb640" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59e4d44abd324614878413675cd9ece6"
I20260924 21:08:05.630430 1801 raft_consensus.cc:3037] T 94695eeed80d43e1b06988cf07fa27ff P 59e4d44abd324614878413675cd9ece6 [term 0 FOLLOWER]: Advancing to term 1
I20260924 21:08:05.630450 1661 raft_consensus.cc:3037] T 94695eeed80d43e1b06988cf07fa27ff P c7395ba0e4ad43ea965a4a9847899e4e [term 0 FOLLOWER]: Advancing to term 1
I20260924 21:08:05.630999 1661 raft_consensus.cc:2454] T 94695eeed80d43e1b06988cf07fa27ff P c7395ba0e4ad43ea965a4a9847899e4e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f896991b69e44a1bbb7b271911bdb640 in term 1.
I20260924 21:08:05.631013 1801 raft_consensus.cc:2454] T 94695eeed80d43e1b06988cf07fa27ff P 59e4d44abd324614878413675cd9ece6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f896991b69e44a1bbb7b271911bdb640 in term 1.
I20260924 21:08:05.631155 1872 leader_election.cc:302] T 94695eeed80d43e1b06988cf07fa27ff P f896991b69e44a1bbb7b271911bdb640 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c7395ba0e4ad43ea965a4a9847899e4e, f896991b69e44a1bbb7b271911bdb640; no voters:
I20260924 21:08:05.631268 2017 raft_consensus.cc:2781] T 94695eeed80d43e1b06988cf07fa27ff P f896991b69e44a1bbb7b271911bdb640 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 21:08:05.631325 2017 raft_consensus.cc:686] T 94695eeed80d43e1b06988cf07fa27ff P f896991b69e44a1bbb7b271911bdb640 [term 1 LEADER]: Becoming Leader. State: Replica: f896991b69e44a1bbb7b271911bdb640, State: Running, Role: LEADER
I20260924 21:08:05.631359 2017 consensus_queue.cc:237] T 94695eeed80d43e1b06988cf07fa27ff P f896991b69e44a1bbb7b271911bdb640 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7395ba0e4ad43ea965a4a9847899e4e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43729 } } peers { permanent_uuid: "59e4d44abd324614878413675cd9ece6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45399 } } peers { permanent_uuid: "f896991b69e44a1bbb7b271911bdb640" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36597 } }
I20260924 21:08:05.631634 2011 tablet_replica.cc:439] T 94695eeed80d43e1b06988cf07fa27ff P f896991b69e44a1bbb7b271911bdb640: TxnStatusTablet state changed. Reason: New leader f896991b69e44a1bbb7b271911bdb640. Latest consensus state: current_term: 1 leader_uuid: "f896991b69e44a1bbb7b271911bdb640" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7395ba0e4ad43ea965a4a9847899e4e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43729 } } peers { permanent_uuid: "59e4d44abd324614878413675cd9ece6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45399 } } peers { permanent_uuid: "f896991b69e44a1bbb7b271911bdb640" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36597 } } }
I20260924 21:08:05.631707 2011 tablet_replica.cc:442] T 94695eeed80d43e1b06988cf07fa27ff P f896991b69e44a1bbb7b271911bdb640: This TxnStatusTablet replica's current role is: LEADER
I20260924 21:08:05.631886 2049 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260924 21:08:05.631994 2049 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260924 21:08:05.631953 1504 catalog_manager.cc:5495] T 94695eeed80d43e1b06988cf07fa27ff P f896991b69e44a1bbb7b271911bdb640 reported cstate change: term changed from 0 to 1, leader changed from <none> to f896991b69e44a1bbb7b271911bdb640 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "f896991b69e44a1bbb7b271911bdb640" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7395ba0e4ad43ea965a4a9847899e4e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43729 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "59e4d44abd324614878413675cd9ece6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45399 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f896991b69e44a1bbb7b271911bdb640" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36597 } health_report { overall_health: HEALTHY } } }
I20260924 21:08:05.650091 2005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:05.648090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38892 (local address 127.0.98.3:36597)
0924 21:08:05.648138 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0924 21:08:05.648140 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 21:08:05.648151 (+ 11us) server_negotiation.cc:407] Connection header received
0924 21:08:05.648227 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:08:05.648228 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:08:05.648239 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:08:05.648286 (+ 47us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 21:08:05.648379 (+ 93us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:05.648782 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:05.649467 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:05.649672 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:05.649734 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:05.649894 (+ 160us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 21:08:05.649922 (+ 28us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 21:08:05.649967 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0924 21:08:05.650021 (+ 54us) server_negotiation.cc:299] Negotiation successful
0924 21:08:05.650038 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260924 21:08:05.650123 2044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:05.648038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36597 (local address 127.0.0.1:38892)
0924 21:08:05.648113 (+ 75us) negotiation.cc:107] Waiting for socket to connect
0924 21:08:05.648124 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 21:08:05.648167 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 21:08:05.648290 (+ 123us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 21:08:05.648291 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 21:08:05.648292 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 21:08:05.648335 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:05.648337 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:05.648869 (+ 532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:05.648870 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:05.649359 (+ 489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:05.649360 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:05.649733 (+ 373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:05.649734 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:05.649790 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:05.649819 (+ 29us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 21:08:05.649969 (+ 150us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 21:08:05.649975 (+ 6us) client_negotiation.cc:769] Sending connection context
0924 21:08:05.650018 (+ 43us) client_negotiation.cc:240] Negotiation successful
0924 21:08:05.650058 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260924 21:08:05.651057 1801 raft_consensus.cc:1261] T 94695eeed80d43e1b06988cf07fa27ff P 59e4d44abd324614878413675cd9ece6 [term 1 FOLLOWER]: Refusing update from remote peer f896991b69e44a1bbb7b271911bdb640: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 21:08:05.651100 1661 raft_consensus.cc:1261] T 94695eeed80d43e1b06988cf07fa27ff P c7395ba0e4ad43ea965a4a9847899e4e [term 1 FOLLOWER]: Refusing update from remote peer f896991b69e44a1bbb7b271911bdb640: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 21:08:05.651258 2011 consensus_queue.cc:1035] T 94695eeed80d43e1b06988cf07fa27ff P f896991b69e44a1bbb7b271911bdb640 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c7395ba0e4ad43ea965a4a9847899e4e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43729 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 21:08:05.651335 2017 consensus_queue.cc:1035] T 94695eeed80d43e1b06988cf07fa27ff P f896991b69e44a1bbb7b271911bdb640 [LEADER]: Connected to new peer: Peer: permanent_uuid: "59e4d44abd324614878413675cd9ece6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 21:08:05.651834 2015 tablet_replica.cc:439] T 94695eeed80d43e1b06988cf07fa27ff P 59e4d44abd324614878413675cd9ece6: TxnStatusTablet state changed. Reason: New leader f896991b69e44a1bbb7b271911bdb640. Latest consensus state: current_term: 1 leader_uuid: "f896991b69e44a1bbb7b271911bdb640" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7395ba0e4ad43ea965a4a9847899e4e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43729 } } peers { permanent_uuid: "59e4d44abd324614878413675cd9ece6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45399 } } peers { permanent_uuid: "f896991b69e44a1bbb7b271911bdb640" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36597 } } }
I20260924 21:08:05.651918 2015 tablet_replica.cc:442] T 94695eeed80d43e1b06988cf07fa27ff P 59e4d44abd324614878413675cd9ece6: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 21:08:05.652259 2013 tablet_replica.cc:439] T 94695eeed80d43e1b06988cf07fa27ff P c7395ba0e4ad43ea965a4a9847899e4e: TxnStatusTablet state changed. Reason: New leader f896991b69e44a1bbb7b271911bdb640. Latest consensus state: current_term: 1 leader_uuid: "f896991b69e44a1bbb7b271911bdb640" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7395ba0e4ad43ea965a4a9847899e4e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43729 } } peers { permanent_uuid: "59e4d44abd324614878413675cd9ece6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45399 } } peers { permanent_uuid: "f896991b69e44a1bbb7b271911bdb640" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36597 } } }
I20260924 21:08:05.652331 2013 tablet_replica.cc:442] T 94695eeed80d43e1b06988cf07fa27ff P c7395ba0e4ad43ea965a4a9847899e4e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 21:08:05.652568 2011 tablet_replica.cc:439] T 94695eeed80d43e1b06988cf07fa27ff P f896991b69e44a1bbb7b271911bdb640: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f896991b69e44a1bbb7b271911bdb640" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7395ba0e4ad43ea965a4a9847899e4e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43729 } } peers { permanent_uuid: "59e4d44abd324614878413675cd9ece6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45399 } } peers { permanent_uuid: "f896991b69e44a1bbb7b271911bdb640" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36597 } } }
I20260924 21:08:05.652633 2011 tablet_replica.cc:442] T 94695eeed80d43e1b06988cf07fa27ff P f896991b69e44a1bbb7b271911bdb640: This TxnStatusTablet replica's current role is: LEADER
I20260924 21:08:05.653055 2017 tablet_replica.cc:439] T 94695eeed80d43e1b06988cf07fa27ff P f896991b69e44a1bbb7b271911bdb640: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f896991b69e44a1bbb7b271911bdb640" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7395ba0e4ad43ea965a4a9847899e4e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43729 } } peers { permanent_uuid: "59e4d44abd324614878413675cd9ece6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45399 } } peers { permanent_uuid: "f896991b69e44a1bbb7b271911bdb640" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36597 } } }
I20260924 21:08:05.653120 2017 tablet_replica.cc:442] T 94695eeed80d43e1b06988cf07fa27ff P f896991b69e44a1bbb7b271911bdb640: This TxnStatusTablet replica's current role is: LEADER
I20260924 21:08:05.653080 2015 tablet_replica.cc:439] T 94695eeed80d43e1b06988cf07fa27ff P 59e4d44abd324614878413675cd9ece6: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f896991b69e44a1bbb7b271911bdb640" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7395ba0e4ad43ea965a4a9847899e4e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43729 } } peers { permanent_uuid: "59e4d44abd324614878413675cd9ece6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45399 } } peers { permanent_uuid: "f896991b69e44a1bbb7b271911bdb640" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36597 } } }
I20260924 21:08:05.653142 2015 tablet_replica.cc:442] T 94695eeed80d43e1b06988cf07fa27ff P 59e4d44abd324614878413675cd9ece6: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 21:08:05.653400 2013 tablet_replica.cc:439] T 94695eeed80d43e1b06988cf07fa27ff P c7395ba0e4ad43ea965a4a9847899e4e: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f896991b69e44a1bbb7b271911bdb640" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7395ba0e4ad43ea965a4a9847899e4e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43729 } } peers { permanent_uuid: "59e4d44abd324614878413675cd9ece6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45399 } } peers { permanent_uuid: "f896991b69e44a1bbb7b271911bdb640" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36597 } } }
I20260924 21:08:05.653465 2013 tablet_replica.cc:442] T 94695eeed80d43e1b06988cf07fa27ff P c7395ba0e4ad43ea965a4a9847899e4e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 21:08:05.656690 2006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:05.654641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40142 (local address 127.0.98.2:45399)
0924 21:08:05.654681 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0924 21:08:05.654683 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 21:08:05.654691 (+ 8us) server_negotiation.cc:407] Connection header received
0924 21:08:05.654761 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:08:05.654762 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:08:05.654779 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:08:05.654846 (+ 67us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 21:08:05.654947 (+ 101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:05.655384 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:05.656038 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:05.656222 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:05.656285 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:05.656395 (+ 110us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 21:08:05.656486 (+ 91us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 21:08:05.656555 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0924 21:08:05.656589 (+ 34us) server_negotiation.cc:299] Negotiation successful
0924 21:08:05.656607 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260924 21:08:05.661212 2005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:05.659022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:43515 (local address 127.0.98.3:36597)
0924 21:08:05.659058 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0924 21:08:05.659059 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 21:08:05.659120 (+ 61us) server_negotiation.cc:407] Connection header received
0924 21:08:05.659179 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:08:05.659180 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:08:05.659187 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:08:05.659232 (+ 45us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 21:08:05.659336 (+ 104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:05.659690 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:05.660607 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:05.660778 (+ 171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:05.660834 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:05.661033 (+ 199us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 21:08:05.661054 (+ 21us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 21:08:05.661089 (+ 35us) server_negotiation.cc:1035] Waiting for connection context
0924 21:08:05.661158 (+ 69us) server_negotiation.cc:299] Negotiation successful
0924 21:08:05.661174 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260924 21:08:05.661270 2055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:05.658965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36597 (local address 127.0.98.2:43515)
0924 21:08:05.659074 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0924 21:08:05.659082 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0924 21:08:05.659131 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 21:08:05.659233 (+ 102us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 21:08:05.659234 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 21:08:05.659235 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 21:08:05.659289 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:05.659290 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:05.659751 (+ 461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:05.659752 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:05.660524 (+ 772us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:05.660526 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:05.660863 (+ 337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:05.660864 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:05.660948 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:05.660960 (+ 12us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 21:08:05.661107 (+ 147us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 21:08:05.661109 (+ 2us) client_negotiation.cc:769] Sending connection context
0924 21:08:05.661157 (+ 48us) client_negotiation.cc:240] Negotiation successful
0924 21:08:05.661196 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":80,"thread_start_us":60,"threads_started":1}
I20260924 21:08:05.663098 1661 raft_consensus.cc:1261] T a3a0a564d3a14700a68ea68427f338bf P c7395ba0e4ad43ea965a4a9847899e4e [term 1 FOLLOWER]: Refusing update from remote peer 59e4d44abd324614878413675cd9ece6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 21:08:05.663175 1941 raft_consensus.cc:1261] T a3a0a564d3a14700a68ea68427f338bf P f896991b69e44a1bbb7b271911bdb640 [term 1 FOLLOWER]: Refusing update from remote peer 59e4d44abd324614878413675cd9ece6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 21:08:05.663530 2037 consensus_queue.cc:1035] T a3a0a564d3a14700a68ea68427f338bf P 59e4d44abd324614878413675cd9ece6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f896991b69e44a1bbb7b271911bdb640" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36597 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 21:08:05.663686 2037 consensus_queue.cc:1035] T a3a0a564d3a14700a68ea68427f338bf P 59e4d44abd324614878413675cd9ece6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c7395ba0e4ad43ea965a4a9847899e4e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43729 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 21:08:05.664646 2021 mvcc.cc:205] Tried to move back new op lower bound from 7333003614874525696 to 7333003614270459904. Current Snapshot: MvccSnapshot[applied={T|T < 7333003614874525696}]
I20260924 21:08:05.673530 2006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:05.671225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:58061 (local address 127.0.98.2:45399)
0924 21:08:05.671282 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0924 21:08:05.671284 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 21:08:05.671294 (+ 10us) server_negotiation.cc:407] Connection header received
0924 21:08:05.671361 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:08:05.671365 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:08:05.671384 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:08:05.671457 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 21:08:05.671563 (+ 106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:05.671958 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:05.672943 (+ 985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:05.673198 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:05.673253 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:05.673384 (+ 131us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 21:08:05.673404 (+ 20us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 21:08:05.673432 (+ 28us) server_negotiation.cc:1035] Waiting for connection context
0924 21:08:05.673478 (+ 46us) server_negotiation.cc:299] Negotiation successful
0924 21:08:05.673490 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260924 21:08:05.673581 2060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:05.671089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45399 (local address 127.0.98.3:58061)
0924 21:08:05.671258 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0924 21:08:05.671266 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0924 21:08:05.671306 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 21:08:05.671463 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 21:08:05.671463 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 21:08:05.671465 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 21:08:05.671506 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:05.671507 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:05.672017 (+ 510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:05.672018 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:05.672807 (+ 789us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:05.672809 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:05.673249 (+ 440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:05.673250 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:05.673299 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:05.673311 (+ 12us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 21:08:05.673434 (+ 123us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 21:08:05.673435 (+ 1us) client_negotiation.cc:769] Sending connection context
0924 21:08:05.673476 (+ 41us) client_negotiation.cc:240] Negotiation successful
0924 21:08:05.673511 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":60,"threads_started":1}
I20260924 21:08:05.685889 1503 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41020:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 21:08:05.685966 1503 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41020:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 21:08:05.686519 1503 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ecbe9754db8046bea8afe31efa3a107a: Sending DeleteTablet for 3 replicas of tablet a3a0a564d3a14700a68ea68427f338bf
I20260924 21:08:05.686761 1641 tablet_service.cc:1514] Processing DeleteTablet for tablet a3a0a564d3a14700a68ea68427f338bf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 21:08:05 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:50404
I20260924 21:08:05.686791 1921 tablet_service.cc:1514] Processing DeleteTablet for tablet a3a0a564d3a14700a68ea68427f338bf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 21:08:05 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:38864
I20260924 21:08:05.686885 2029 tablet_replica.cc:330] T a3a0a564d3a14700a68ea68427f338bf P c7395ba0e4ad43ea965a4a9847899e4e: stopping tablet replica
I20260924 21:08:05.686939 2029 raft_consensus.cc:2229] T a3a0a564d3a14700a68ea68427f338bf P c7395ba0e4ad43ea965a4a9847899e4e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 21:08:05.686969 2029 raft_consensus.cc:2258] T a3a0a564d3a14700a68ea68427f338bf P c7395ba0e4ad43ea965a4a9847899e4e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 21:08:05.687033 2030 tablet_replica.cc:330] T a3a0a564d3a14700a68ea68427f338bf P f896991b69e44a1bbb7b271911bdb640: stopping tablet replica
I20260924 21:08:05.687081 2030 raft_consensus.cc:2229] T a3a0a564d3a14700a68ea68427f338bf P f896991b69e44a1bbb7b271911bdb640 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 21:08:05.687114 2030 raft_consensus.cc:2258] T a3a0a564d3a14700a68ea68427f338bf P f896991b69e44a1bbb7b271911bdb640 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 21:08:05.687394 1781 tablet_service.cc:1514] Processing DeleteTablet for tablet a3a0a564d3a14700a68ea68427f338bf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 21:08:05 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:40130
I20260924 21:08:05.687440 2030 ts_tablet_manager.cc:1905] T a3a0a564d3a14700a68ea68427f338bf P f896991b69e44a1bbb7b271911bdb640: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 21:08:05.687469 2031 tablet_replica.cc:330] T a3a0a564d3a14700a68ea68427f338bf P 59e4d44abd324614878413675cd9ece6: stopping tablet replica
I20260924 21:08:05.687501 2031 raft_consensus.cc:2229] T a3a0a564d3a14700a68ea68427f338bf P 59e4d44abd324614878413675cd9ece6 [term 1 LEADER]: Raft consensus shutting down.
I20260924 21:08:05.687566 2031 raft_consensus.cc:2258] T a3a0a564d3a14700a68ea68427f338bf P 59e4d44abd324614878413675cd9ece6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 21:08:05.687836 2031 ts_tablet_manager.cc:1905] T a3a0a564d3a14700a68ea68427f338bf P 59e4d44abd324614878413675cd9ece6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 21:08:05.689046 2030 ts_tablet_manager.cc:1918] T a3a0a564d3a14700a68ea68427f338bf P f896991b69e44a1bbb7b271911bdb640: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 21:08:05.689103 2030 log.cc:1196] T a3a0a564d3a14700a68ea68427f338bf P f896991b69e44a1bbb7b271911bdb640: Deleting WAL directory at /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790284076800039-392-0/minicluster-data/ts-2/wal/wals/a3a0a564d3a14700a68ea68427f338bf
I20260924 21:08:05.689190 2031 ts_tablet_manager.cc:1918] T a3a0a564d3a14700a68ea68427f338bf P 59e4d44abd324614878413675cd9ece6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 21:08:05.689242 2031 log.cc:1196] T a3a0a564d3a14700a68ea68427f338bf P 59e4d44abd324614878413675cd9ece6: Deleting WAL directory at /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790284076800039-392-0/minicluster-data/ts-1/wal/wals/a3a0a564d3a14700a68ea68427f338bf
I20260924 21:08:05.689316 2030 ts_tablet_manager.cc:1939] T a3a0a564d3a14700a68ea68427f338bf P f896991b69e44a1bbb7b271911bdb640: Deleting consensus metadata
I20260924 21:08:05.689422 2031 ts_tablet_manager.cc:1939] T a3a0a564d3a14700a68ea68427f338bf P 59e4d44abd324614878413675cd9ece6: Deleting consensus metadata
I20260924 21:08:05.689631 1491 catalog_manager.cc:4841] TS f896991b69e44a1bbb7b271911bdb640 (127.0.98.3:36597): tablet a3a0a564d3a14700a68ea68427f338bf (table test-table [id=d3bd332c0ab841cf95204839b352a118]) successfully deleted
I20260924 21:08:05.689723 2029 ts_tablet_manager.cc:1905] T a3a0a564d3a14700a68ea68427f338bf P c7395ba0e4ad43ea965a4a9847899e4e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 21:08:05.689929 1489 catalog_manager.cc:4841] TS 59e4d44abd324614878413675cd9ece6 (127.0.98.2:45399): tablet a3a0a564d3a14700a68ea68427f338bf (table test-table [id=d3bd332c0ab841cf95204839b352a118]) successfully deleted
I20260924 21:08:05.691154 2029 ts_tablet_manager.cc:1918] T a3a0a564d3a14700a68ea68427f338bf P c7395ba0e4ad43ea965a4a9847899e4e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 21:08:05.691200 2029 log.cc:1196] T a3a0a564d3a14700a68ea68427f338bf P c7395ba0e4ad43ea965a4a9847899e4e: Deleting WAL directory at /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790284076800039-392-0/minicluster-data/ts-0/wal/wals/a3a0a564d3a14700a68ea68427f338bf
I20260924 21:08:05.691390 2029 ts_tablet_manager.cc:1939] T a3a0a564d3a14700a68ea68427f338bf P c7395ba0e4ad43ea965a4a9847899e4e: Deleting consensus metadata
I20260924 21:08:05.691690 1488 catalog_manager.cc:4841] TS c7395ba0e4ad43ea965a4a9847899e4e (127.0.98.1:43729): tablet a3a0a564d3a14700a68ea68427f338bf (table test-table [id=d3bd332c0ab841cf95204839b352a118]) successfully deleted
Sep 24 21:08:05 dist-test-slave-c7tk krb5kdc[1459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284085, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260924 21:08:05.702908 2004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:05.694059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50408 (local address 127.0.98.1:43729)
0924 21:08:05.694098 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0924 21:08:05.694099 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 21:08:05.694231 (+ 132us) server_negotiation.cc:407] Connection header received
0924 21:08:05.694260 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:08:05.694261 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:08:05.694272 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:08:05.694338 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:08:05.694965 (+ 627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:05.695560 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:05.696446 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:05.696756 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:05.696837 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:05.699812 (+ 2975us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 21:08:05.699819 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 21:08:05.699819 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 21:08:05.699839 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0924 21:08:05.701703 (+ 1864us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:08:05.702095 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:08:05.702098 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:08:05.702098 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:08:05.702174 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:08:05.702382 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:08:05.702382 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:08:05.702383 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:08:05.702530 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 21:08:05.702660 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0924 21:08:05.702744 (+ 84us) server_negotiation.cc:299] Negotiation successful
0924 21:08:05.702817 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
W20260924 21:08:05.703423 1701 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50408
I20260924 21:08:05.711050 1553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:05.704308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41038 (local address 127.0.98.62:39063)
0924 21:08:05.704347 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0924 21:08:05.704349 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 21:08:05.704555 (+ 206us) server_negotiation.cc:407] Connection header received
0924 21:08:05.704599 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:08:05.704600 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:08:05.704611 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:08:05.704674 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:08:05.705218 (+ 544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:05.705877 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:05.706767 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:05.707070 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:05.707148 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:05.708401 (+ 1253us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 21:08:05.708402 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 21:08:05.708402 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 21:08:05.708409 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 21:08:05.709928 (+ 1519us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:08:05.710288 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:08:05.710289 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:08:05.710290 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:08:05.710344 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:08:05.710541 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:08:05.710542 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:08:05.710542 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:08:05.710676 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 21:08:05.710775 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0924 21:08:05.710906 (+ 131us) server_negotiation.cc:299] Negotiation successful
0924 21:08:05.710974 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
W20260924 21:08:05.711532 1503 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41038
I20260924 21:08:05.712169 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAzNNjS/build/release/bin/kudu with pid 1576
I20260924 21:08:05.719255 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAzNNjS/build/release/bin/kudu with pid 1716
I20260924 21:08:05.724013 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAzNNjS/build/release/bin/kudu with pid 1856
I20260924 21:08:05.729362 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAzNNjS/build/release/bin/kudu with pid 1472
2026-09-24T21:08:05Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (2203 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790284076800039-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 21:08:05 dist-test-slave-c7tk krb5kdc[2069](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 21:08:05 dist-test-slave-c7tk krb5kdc[2069](info): set up 2 sockets
Sep 24 21:08:05 dist-test-slave-c7tk krb5kdc[2069](info): commencing operation
krb5kdc: starting...
W20260924 21:08:06.669478 392 mini_kdc.cc:121] Time spent starting KDC: real 0.920s user 0.004s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790284076800039-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790284076800039-392-0/krb5kdc/test-user.keytab.
Sep 24 21:08:06 dist-test-slave-c7tk krb5kdc[2069](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284086, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T21:08:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T21:08:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790284076800039-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790284076800039-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 21:08:06.780128 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAzNNjS/build/release/bin/kudu
/tmp/dist-test-taskAzNNjS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790284076800039-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790284076800039-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790284076800039-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790284076800039-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33355
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790284076800039-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36479
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33355
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790284076800039-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790284076800039-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790284076800039-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 21:08:06.853613 2082 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 21:08:06.853848 2082 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 21:08:06.853891 2082 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 21:08:06.854981 2082 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 21:08:06.855050 2082 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260924 21:08:06.855074 2082 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 21:08:06.855105 2082 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 21:08:06.855129 2082 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 21:08:06.855156 2082 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 21:08:06.856544 2082 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36479
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790284076800039-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790284076800039-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33355
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790284076800039-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33355
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790284076800039-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2082
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790284076800039-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 19:49:06 UTC on ccycloud.quasar-vhqjoc.root.comops.site
build id 4670864
I20260924 21:08:06.856813 2082 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 21:08:06.857031 2082 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 21:08:06.861553 2090 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 21:08:06.861557 2088 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 21:08:06.861563 2087 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 21:08:06.861671 2082 server_base.cc:1034] running on GCE node
I20260924 21:08:06.861939 2082 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 21:08:06.862197 2082 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 21:08:06.863343 2082 hybrid_clock.cc:648] HybridClock initialized: now 1790284086863333 us; error 30 us; skew 500 ppm
Sep 24 21:08:06 dist-test-slave-c7tk krb5kdc[2069](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284086, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 21:08:06.865578 2082 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 21:08:06.866048 2082 webserver.cc:460] Webserver started at http://127.0.98.62:40359/ using document root <none> and password file <none>
I20260924 21:08:06.866251 2082 fs_manager.cc:359] Metadata directory not provided
I20260924 21:08:06.866293 2082 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 21:08:06.866451 2082 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 21:08:06.867437 2082 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790284076800039-392-0/minicluster-data/master-0/data/instance:
uuid: "224fefe284af480497048ba6af0e88cf"
format_stamp: "Formatted at 2026-09-24 21:08:06 on dist-test-slave-c7tk"
server_key: "8522d1dbfb7cb2bbcea67fc23d4b4a17"
server_key_iv: "58d6f9b0c9e71a1f84dca8a8a95a8f52"
server_key_version: "encryptionkey@0"
I20260924 21:08:06.867764 2082 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790284076800039-392-0/minicluster-data/master-0/wal/instance:
uuid: "224fefe284af480497048ba6af0e88cf"
format_stamp: "Formatted at 2026-09-24 21:08:06 on dist-test-slave-c7tk"
server_key: "8522d1dbfb7cb2bbcea67fc23d4b4a17"
server_key_iv: "58d6f9b0c9e71a1f84dca8a8a95a8f52"
server_key_version: "encryptionkey@0"
I20260924 21:08:06.868956 2082 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 21:08:06.869623 2097 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 21:08:06.869840 2082 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 21:08:06.869938 2082 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790284076800039-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790284076800039-392-0/minicluster-data/master-0/wal
uuid: "224fefe284af480497048ba6af0e88cf"
format_stamp: "Formatted at 2026-09-24 21:08:06 on dist-test-slave-c7tk"
server_key: "8522d1dbfb7cb2bbcea67fc23d4b4a17"
server_key_iv: "58d6f9b0c9e71a1f84dca8a8a95a8f52"
server_key_version: "encryptionkey@0"
I20260924 21:08:06.870016 2082 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790284076800039-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790284076800039-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790284076800039-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 21:08:06.893147 2082 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 21:08:06.894897 2082 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 21:08:06.895049 2082 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 21:08:06.899137 2082 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33355
I20260924 21:08:06.899178 2159 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33355 every 8 connection(s)
I20260924 21:08:06.899480 2082 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790284076800039-392-0/minicluster-data/master-0/data/info.pb
I20260924 21:08:06.900029 2160 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 21:08:06.902133 2160 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 224fefe284af480497048ba6af0e88cf: Bootstrap starting.
I20260924 21:08:06.902697 2160 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 224fefe284af480497048ba6af0e88cf: Neither blocks nor log segments found. Creating new log.
I20260924 21:08:06.902977 2160 log.cc:824] T 00000000000000000000000000000000 P 224fefe284af480497048ba6af0e88cf: Log is configured to *not* fsync() on all Append() calls
I20260924 21:08:06.903611 2160 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 224fefe284af480497048ba6af0e88cf: No bootstrap required, opened a new log
I20260924 21:08:06.904523 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAzNNjS/build/release/bin/kudu as pid 2082
I20260924 21:08:06.904590 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790284076800039-392-0/minicluster-data/master-0/wal/instance
I20260924 21:08:06.904767 392 external_mini_cluster.cc:1428] Setting key af08fbf1d1569891e48c55e81761603d
I20260924 21:08:06.905546 2160 raft_consensus.cc:348] T 00000000000000000000000000000000 P 224fefe284af480497048ba6af0e88cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "224fefe284af480497048ba6af0e88cf" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33355 } }
I20260924 21:08:06.905714 2160 raft_consensus.cc:374] T 00000000000000000000000000000000 P 224fefe284af480497048ba6af0e88cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 21:08:06.905758 2160 raft_consensus.cc:729] T 00000000000000000000000000000000 P 224fefe284af480497048ba6af0e88cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 224fefe284af480497048ba6af0e88cf, State: Initialized, Role: FOLLOWER
I20260924 21:08:06.905890 2160 consensus_queue.cc:260] T 00000000000000000000000000000000 P 224fefe284af480497048ba6af0e88cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "224fefe284af480497048ba6af0e88cf" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33355 } }
I20260924 21:08:06.906162 2160 raft_consensus.cc:388] T 00000000000000000000000000000000 P 224fefe284af480497048ba6af0e88cf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 21:08:06.906221 2160 raft_consensus.cc:482] T 00000000000000000000000000000000 P 224fefe284af480497048ba6af0e88cf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 21:08:06.906275 2160 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 224fefe284af480497048ba6af0e88cf [term 0 FOLLOWER]: Advancing to term 1
I20260924 21:08:06.906876 2160 raft_consensus.cc:504] T 00000000000000000000000000000000 P 224fefe284af480497048ba6af0e88cf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "224fefe284af480497048ba6af0e88cf" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33355 } }
I20260924 21:08:06.907008 2160 leader_election.cc:302] T 00000000000000000000000000000000 P 224fefe284af480497048ba6af0e88cf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 224fefe284af480497048ba6af0e88cf; no voters:
I20260924 21:08:06.907202 2160 leader_election.cc:288] T 00000000000000000000000000000000 P 224fefe284af480497048ba6af0e88cf [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 21:08:06.907249 2165 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 224fefe284af480497048ba6af0e88cf [term 1 FOLLOWER]: Leader election won for term 1
I20260924 21:08:06.907399 2165 raft_consensus.cc:686] T 00000000000000000000000000000000 P 224fefe284af480497048ba6af0e88cf [term 1 LEADER]: Becoming Leader. State: Replica: 224fefe284af480497048ba6af0e88cf, State: Running, Role: LEADER
I20260924 21:08:06.907469 2160 sys_catalog.cc:564] T 00000000000000000000000000000000 P 224fefe284af480497048ba6af0e88cf [sys.catalog]: configured and running, proceeding with master startup.
I20260924 21:08:06.907478 2165 consensus_queue.cc:237] T 00000000000000000000000000000000 P 224fefe284af480497048ba6af0e88cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "224fefe284af480497048ba6af0e88cf" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33355 } }
I20260924 21:08:06.907833 2165 sys_catalog.cc:455] T 00000000000000000000000000000000 P 224fefe284af480497048ba6af0e88cf [sys.catalog]: SysCatalogTable state changed. Reason: New leader 224fefe284af480497048ba6af0e88cf. Latest consensus state: current_term: 1 leader_uuid: "224fefe284af480497048ba6af0e88cf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "224fefe284af480497048ba6af0e88cf" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33355 } } }
I20260924 21:08:06.907893 2165 sys_catalog.cc:458] T 00000000000000000000000000000000 P 224fefe284af480497048ba6af0e88cf [sys.catalog]: This master's current role is: LEADER
I20260924 21:08:06.907847 2167 sys_catalog.cc:455] T 00000000000000000000000000000000 P 224fefe284af480497048ba6af0e88cf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "224fefe284af480497048ba6af0e88cf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "224fefe284af480497048ba6af0e88cf" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33355 } } }
I20260924 21:08:06.908095 2171 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 21:08:06.908109 2167 sys_catalog.cc:458] T 00000000000000000000000000000000 P 224fefe284af480497048ba6af0e88cf [sys.catalog]: This master's current role is: LEADER
I20260924 21:08:06.908582 2171 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 21:08:06.910286 2171 catalog_manager.cc:1347] Generated new cluster ID: 39ea614524214d7ea717e9ae4c10c21e
I20260924 21:08:06.910342 2171 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 24 21:08:06 dist-test-slave-c7tk krb5kdc[2069](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284086, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 21:08:06.916584 2164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:06.905472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57936 (local address 127.0.98.62:33355)
0924 21:08:06.905644 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0924 21:08:06.905651 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 21:08:06.905672 (+ 21us) server_negotiation.cc:407] Connection header received
0924 21:08:06.905784 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:08:06.905791 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:08:06.905891 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:08:06.906035 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:08:06.906699 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:06.907999 (+ 1300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:06.909093 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:06.909667 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:06.909770 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:06.912479 (+ 2709us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 21:08:06.912483 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 21:08:06.912483 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 21:08:06.912495 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 21:08:06.914519 (+ 2024us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:08:06.914899 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:08:06.914902 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:08:06.914902 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:08:06.915520 (+ 618us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:08:06.915727 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:08:06.915728 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:08:06.915728 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:08:06.916029 (+ 301us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 21:08:06.916192 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0924 21:08:06.916266 (+ 74us) server_negotiation.cc:299] Negotiation successful
0924 21:08:06.916397 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":79,"threads_started":1}
I20260924 21:08:06.933807 2171 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 21:08:06.934453 2171 catalog_manager.cc:1504] Loading token signing keys...
I20260924 21:08:06.941812 2171 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 224fefe284af480497048ba6af0e88cf: Generated new TSK 0
I20260924 21:08:06.941977 2171 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790284076800039-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790284076800039-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260924 21:08:06.971679 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAzNNjS/build/release/bin/kudu
/tmp/dist-test-taskAzNNjS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790284076800039-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790284076800039-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790284076800039-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790284076800039-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33355
--keytab_file=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790284076800039-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36479
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790284076800039-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790284076800039-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790284076800039-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 21:08:07.041772 2186 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 21:08:07.041942 2186 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 21:08:07.041962 2186 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 21:08:07.042889 2186 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 21:08:07.042919 2186 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 21:08:07.042937 2186 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260924 21:08:07.042948 2186 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 21:08:07.042963 2186 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 21:08:07.044116 2186 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36479
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790284076800039-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790284076800039-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790284076800039-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790284076800039-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:33355
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2186
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790284076800039-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 19:49:06 UTC on ccycloud.quasar-vhqjoc.root.comops.site
build id 4670864
I20260924 21:08:07.044250 2186 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 21:08:07.044394 2186 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 21:08:07.048700 2192 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 21:08:07.048753 2191 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 21:08:07.048853 2186 server_base.cc:1034] running on GCE node
W20260924 21:08:07.048734 2194 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 21:08:07.049120 2186 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 21:08:07.049340 2186 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 21:08:07.050472 2186 hybrid_clock.cc:648] HybridClock initialized: now 1790284087050457 us; error 26 us; skew 500 ppm
Sep 24 21:08:07 dist-test-slave-c7tk krb5kdc[2069](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284087, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 21:08:07.052627 2186 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 21:08:07.052985 2186 webserver.cc:460] Webserver started at http://127.0.98.1:35475/ using document root <none> and password file <none>
I20260924 21:08:07.053128 2186 fs_manager.cc:359] Metadata directory not provided
I20260924 21:08:07.053162 2186 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 21:08:07.053233 2186 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 21:08:07.054107 2186 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790284076800039-392-0/minicluster-data/ts-0/data/instance:
uuid: "508ed3a686554bd6af0204840904ab09"
format_stamp: "Formatted at 2026-09-24 21:08:07 on dist-test-slave-c7tk"
server_key: "e3ece49c87dbca512034c7eb175db053"
server_key_iv: "046eaf48296190dba71b6e392c9ec1cf"
server_key_version: "encryptionkey@0"
I20260924 21:08:07.054401 2186 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790284076800039-392-0/minicluster-data/ts-0/wal/instance:
uuid: "508ed3a686554bd6af0204840904ab09"
format_stamp: "Formatted at 2026-09-24 21:08:07 on dist-test-slave-c7tk"
server_key: "e3ece49c87dbca512034c7eb175db053"
server_key_iv: "046eaf48296190dba71b6e392c9ec1cf"
server_key_version: "encryptionkey@0"
I20260924 21:08:07.055586 2186 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 21:08:07.056141 2201 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 21:08:07.056331 2186 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 21:08:07.056408 2186 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790284076800039-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790284076800039-392-0/minicluster-data/ts-0/wal
uuid: "508ed3a686554bd6af0204840904ab09"
format_stamp: "Formatted at 2026-09-24 21:08:07 on dist-test-slave-c7tk"
server_key: "e3ece49c87dbca512034c7eb175db053"
server_key_iv: "046eaf48296190dba71b6e392c9ec1cf"
server_key_version: "encryptionkey@0"
I20260924 21:08:07.056468 2186 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790284076800039-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790284076800039-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790284076800039-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 21:08:07.077621 2186 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 21:08:07.079558 2186 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 21:08:07.079676 2186 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 21:08:07.080082 2186 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 21:08:07.080137 2186 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 21:08:07.080173 2186 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 21:08:07.080210 2186 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 21:08:07 dist-test-slave-c7tk krb5kdc[2069](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284087, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 21:08:07.086400 2186 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:44497
I20260924 21:08:07.086447 2316 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:44497 every 8 connection(s)
I20260924 21:08:07.086679 2186 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790284076800039-392-0/minicluster-data/ts-0/data/info.pb
I20260924 21:08:07.089032 2164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:07.080500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:48569 (local address 127.0.98.62:33355)
0924 21:08:07.080539 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0924 21:08:07.080541 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 21:08:07.080742 (+ 201us) server_negotiation.cc:407] Connection header received
0924 21:08:07.081143 (+ 401us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:08:07.081145 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:08:07.081159 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:08:07.081220 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:08:07.082020 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:07.082707 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:07.084086 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:07.084283 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:07.084351 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:07.086594 (+ 2243us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 21:08:07.086595 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 21:08:07.086595 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 21:08:07.086602 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 21:08:07.088044 (+ 1442us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:08:07.088365 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:08:07.088365 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:08:07.088366 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:08:07.088418 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:08:07.088572 (+ 154us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:08:07.088572 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:08:07.088572 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:08:07.088656 (+ 84us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 21:08:07.088699 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0924 21:08:07.088912 (+ 213us) server_negotiation.cc:299] Negotiation successful
0924 21:08:07.088974 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260924 21:08:07.089135 2210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:07.080492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33355 (local address 127.0.98.1:48569)
0924 21:08:07.080641 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0924 21:08:07.080668 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0924 21:08:07.081052 (+ 384us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 21:08:07.081324 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 21:08:07.081328 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 21:08:07.081662 (+ 334us) client_negotiation.cc:189] Negotiated authn=SASL
0924 21:08:07.081925 (+ 263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:07.081930 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:07.082793 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:07.082794 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:07.083995 (+ 1201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:07.083998 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:07.084354 (+ 356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:07.084355 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:07.084493 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:07.084909 (+ 416us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 21:08:07.084919 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 21:08:07.086525 (+ 1606us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 21:08:07.088108 (+ 1583us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 21:08:07.088113 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 21:08:07.088114 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 21:08:07.088304 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 21:08:07.088454 (+ 150us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 21:08:07.088455 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 21:08:07.088455 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 21:08:07.088527 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 21:08:07.088706 (+ 179us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 21:08:07.088710 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 21:08:07.088801 (+ 91us) client_negotiation.cc:769] Sending connection context
0924 21:08:07.088872 (+ 71us) client_negotiation.cc:240] Negotiation successful
0924 21:08:07.088971 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":81,"threads_started":1}
I20260924 21:08:07.089534 2317 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33355
I20260924 21:08:07.089622 2317 heartbeater.cc:464] Registering TS with master...
I20260924 21:08:07.089766 2317 heartbeater.cc:511] Master 127.0.98.62:33355 requested a full tablet report, sending...
I20260924 21:08:07.090240 2114 ts_manager.cc:194] Registered new tserver with Master: 508ed3a686554bd6af0204840904ab09 (127.0.98.1:44497)
I20260924 21:08:07.090895 2114 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:48569
I20260924 21:08:07.094770 2164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:07.090416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:36293 (local address 127.0.98.62:33355)
0924 21:08:07.090456 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0924 21:08:07.090458 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 21:08:07.090637 (+ 179us) server_negotiation.cc:407] Connection header received
0924 21:08:07.090701 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:08:07.090701 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:08:07.090710 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:08:07.090760 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:08:07.091124 (+ 364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:07.091584 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:07.092108 (+ 524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:07.092262 (+ 154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:07.092305 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:07.092943 (+ 638us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 21:08:07.092943 (+ 0us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 21:08:07.092944 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 21:08:07.092949 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 21:08:07.094013 (+ 1064us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:08:07.094289 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:08:07.094289 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:08:07.094290 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:08:07.094335 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:08:07.094458 (+ 123us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:08:07.094458 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:08:07.094458 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:08:07.094535 (+ 77us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 21:08:07.094576 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0924 21:08:07.094653 (+ 77us) server_negotiation.cc:299] Negotiation successful
0924 21:08:07.094708 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260924 21:08:07.094777 2323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:07.090455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33355 (local address 127.0.98.1:36293)
0924 21:08:07.090577 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0924 21:08:07.090585 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0924 21:08:07.090649 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 21:08:07.090763 (+ 114us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 21:08:07.090764 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 21:08:07.090966 (+ 202us) client_negotiation.cc:189] Negotiated authn=SASL
0924 21:08:07.091020 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:07.091022 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:07.091641 (+ 619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:07.091642 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:07.092048 (+ 406us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:07.092049 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:07.092316 (+ 267us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:07.092317 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:07.092361 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:07.092648 (+ 287us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 21:08:07.092651 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0924 21:08:07.092865 (+ 214us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 21:08:07.094076 (+ 1211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 21:08:07.094076 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 21:08:07.094077 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 21:08:07.094228 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 21:08:07.094384 (+ 156us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 21:08:07.094385 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 21:08:07.094385 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 21:08:07.094425 (+ 40us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 21:08:07.094568 (+ 143us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 21:08:07.094568 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 21:08:07.094596 (+ 28us) client_negotiation.cc:769] Sending connection context
0924 21:08:07.094640 (+ 44us) client_negotiation.cc:240] Negotiation successful
0924 21:08:07.094703 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":71,"threads_started":1}
I20260924 21:08:07.095690 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAzNNjS/build/release/bin/kudu as pid 2186
I20260924 21:08:07.095778 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790284076800039-392-0/minicluster-data/ts-0/wal/instance
I20260924 21:08:07.095942 392 external_mini_cluster.cc:1428] Setting key c9c6ceb6adf1e07b0a1eedc13d779a79
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790284076800039-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790284076800039-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260924 21:08:07.117952 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAzNNjS/build/release/bin/kudu
/tmp/dist-test-taskAzNNjS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790284076800039-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790284076800039-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790284076800039-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790284076800039-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33355
--keytab_file=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790284076800039-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36479
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790284076800039-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790284076800039-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790284076800039-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 21:08:07.186434 2326 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 21:08:07.186612 2326 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 21:08:07.186645 2326 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 21:08:07.187767 2326 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 21:08:07.187827 2326 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 21:08:07.187862 2326 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260924 21:08:07.187915 2326 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 21:08:07.187945 2326 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 21:08:07.189217 2326 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36479
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790284076800039-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790284076800039-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790284076800039-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790284076800039-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:33355
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2326
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790284076800039-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 19:49:06 UTC on ccycloud.quasar-vhqjoc.root.comops.site
build id 4670864
I20260924 21:08:07.189424 2326 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 21:08:07.189630 2326 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 21:08:07.194082 2334 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 21:08:07.194089 2331 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 21:08:07.194255 2326 server_base.cc:1034] running on GCE node
W20260924 21:08:07.194095 2332 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 21:08:07.194455 2326 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 21:08:07.194666 2326 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 21:08:07.195809 2326 hybrid_clock.cc:648] HybridClock initialized: now 1790284087195787 us; error 33 us; skew 500 ppm
Sep 24 21:08:07 dist-test-slave-c7tk krb5kdc[2069](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284087, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 21:08:07.198184 2326 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 21:08:07.198556 2326 webserver.cc:460] Webserver started at http://127.0.98.2:36493/ using document root <none> and password file <none>
I20260924 21:08:07.198760 2326 fs_manager.cc:359] Metadata directory not provided
I20260924 21:08:07.198825 2326 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 21:08:07.198982 2326 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 21:08:07.199918 2326 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790284076800039-392-0/minicluster-data/ts-1/data/instance:
uuid: "7a5f5a1ce0964d918460512bd4a094d1"
format_stamp: "Formatted at 2026-09-24 21:08:07 on dist-test-slave-c7tk"
server_key: "16e53160aeb7e327745d11a778656ad7"
server_key_iv: "33b0c8ed7f99a1a41006a8166d4118ce"
server_key_version: "encryptionkey@0"
I20260924 21:08:07.200212 2326 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790284076800039-392-0/minicluster-data/ts-1/wal/instance:
uuid: "7a5f5a1ce0964d918460512bd4a094d1"
format_stamp: "Formatted at 2026-09-24 21:08:07 on dist-test-slave-c7tk"
server_key: "16e53160aeb7e327745d11a778656ad7"
server_key_iv: "33b0c8ed7f99a1a41006a8166d4118ce"
server_key_version: "encryptionkey@0"
I20260924 21:08:07.201277 2326 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 21:08:07.201857 2341 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 21:08:07.202004 2326 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 21:08:07.202081 2326 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790284076800039-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790284076800039-392-0/minicluster-data/ts-1/wal
uuid: "7a5f5a1ce0964d918460512bd4a094d1"
format_stamp: "Formatted at 2026-09-24 21:08:07 on dist-test-slave-c7tk"
server_key: "16e53160aeb7e327745d11a778656ad7"
server_key_iv: "33b0c8ed7f99a1a41006a8166d4118ce"
server_key_version: "encryptionkey@0"
I20260924 21:08:07.202153 2326 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790284076800039-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790284076800039-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790284076800039-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 21:08:07.230968 2326 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 21:08:07.232824 2326 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 21:08:07.233011 2326 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 21:08:07.233443 2326 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 21:08:07.233493 2326 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 21:08:07.233526 2326 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 21:08:07.233546 2326 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 21:08:07 dist-test-slave-c7tk krb5kdc[2069](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284087, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 21:08:07.239899 2326 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:42367
I20260924 21:08:07.240217 2326 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790284076800039-392-0/minicluster-data/ts-1/data/info.pb
I20260924 21:08:07.240398 2456 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:42367 every 8 connection(s)
I20260924 21:08:07.241344 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAzNNjS/build/release/bin/kudu as pid 2326
I20260924 21:08:07.241431 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790284076800039-392-0/minicluster-data/ts-1/wal/instance
I20260924 21:08:07.241600 392 external_mini_cluster.cc:1428] Setting key 3ccf1b4a849dc90d5e773b8d524f40fd
I20260924 21:08:07.242986 2164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:07.234074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:57025 (local address 127.0.98.62:33355)
0924 21:08:07.234128 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0924 21:08:07.234130 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 21:08:07.234337 (+ 207us) server_negotiation.cc:407] Connection header received
0924 21:08:07.234764 (+ 427us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:08:07.234766 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:08:07.234778 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:08:07.234830 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:08:07.235584 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:07.236320 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:07.237833 (+ 1513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:07.238072 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:07.238143 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:07.240309 (+ 2166us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 21:08:07.240311 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 21:08:07.240311 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 21:08:07.240318 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 21:08:07.241697 (+ 1379us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:08:07.242137 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:08:07.242138 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:08:07.242138 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:08:07.242199 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:08:07.242461 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:08:07.242462 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:08:07.242462 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:08:07.242551 (+ 89us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 21:08:07.242597 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0924 21:08:07.242876 (+ 279us) server_negotiation.cc:299] Negotiation successful
0924 21:08:07.242931 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260924 21:08:07.243232 2350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:07.234076 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33355 (local address 127.0.98.2:57025)
0924 21:08:07.234237 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0924 21:08:07.234265 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0924 21:08:07.234663 (+ 398us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 21:08:07.234885 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 21:08:07.234888 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 21:08:07.235234 (+ 346us) client_negotiation.cc:189] Negotiated authn=SASL
0924 21:08:07.235479 (+ 245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:07.235483 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:07.236424 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:07.236425 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:07.237733 (+ 1308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:07.237736 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:07.238145 (+ 409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:07.238146 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:07.238265 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:07.238672 (+ 407us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 21:08:07.238678 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0924 21:08:07.240220 (+ 1542us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 21:08:07.241827 (+ 1607us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 21:08:07.241832 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 21:08:07.241834 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 21:08:07.242049 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 21:08:07.242264 (+ 215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 21:08:07.242264 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 21:08:07.242265 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 21:08:07.242399 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 21:08:07.242609 (+ 210us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 21:08:07.242613 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 21:08:07.242756 (+ 143us) client_negotiation.cc:769] Sending connection context
0924 21:08:07.242855 (+ 99us) client_negotiation.cc:240] Negotiation successful
0924 21:08:07.243002 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":88,"threads_started":1}
I20260924 21:08:07.243836 2457 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33355
I20260924 21:08:07.243937 2457 heartbeater.cc:464] Registering TS with master...
I20260924 21:08:07.244069 2457 heartbeater.cc:511] Master 127.0.98.62:33355 requested a full tablet report, sending...
I20260924 21:08:07.244334 2114 ts_manager.cc:194] Registered new tserver with Master: 7a5f5a1ce0964d918460512bd4a094d1 (127.0.98.2:42367)
I20260924 21:08:07.244736 2114 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:57025
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260924 21:08:07.249715 2164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:07.244751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:50105 (local address 127.0.98.62:33355)
0924 21:08:07.244784 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0924 21:08:07.244786 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 21:08:07.244849 (+ 63us) server_negotiation.cc:407] Connection header received
0924 21:08:07.244911 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:08:07.244912 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:08:07.244919 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:08:07.244965 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:08:07.245228 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:07.245737 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:07.246449 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:07.246641 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:07.246714 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:07.247583 (+ 869us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 21:08:07.247584 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 21:08:07.247584 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 21:08:07.247591 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 21:08:07.248821 (+ 1230us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:08:07.249127 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:08:07.249128 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:08:07.249128 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:08:07.249171 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:08:07.249363 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:08:07.249363 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:08:07.249363 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:08:07.249467 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 21:08:07.249512 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0924 21:08:07.249626 (+ 114us) server_negotiation.cc:299] Negotiation successful
0924 21:08:07.249671 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 21:08:07.249722 2464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:07.244692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33355 (local address 127.0.98.2:50105)
0924 21:08:07.244802 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0924 21:08:07.244811 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 21:08:07.244861 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 21:08:07.244970 (+ 109us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 21:08:07.244970 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 21:08:07.245115 (+ 145us) client_negotiation.cc:189] Negotiated authn=SASL
0924 21:08:07.245157 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:07.245158 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:07.245852 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:07.245853 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:07.246361 (+ 508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:07.246363 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:07.246715 (+ 352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:07.246715 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:07.246784 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:07.247213 (+ 429us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 21:08:07.247218 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 21:08:07.247482 (+ 264us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 21:08:07.248912 (+ 1430us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 21:08:07.248913 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 21:08:07.248913 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 21:08:07.249055 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 21:08:07.249235 (+ 180us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 21:08:07.249236 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 21:08:07.249237 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 21:08:07.249297 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 21:08:07.249531 (+ 234us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 21:08:07.249532 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 21:08:07.249562 (+ 30us) client_negotiation.cc:769] Sending connection context
0924 21:08:07.249608 (+ 46us) client_negotiation.cc:240] Negotiation successful
0924 21:08:07.249667 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":90,"thread_start_us":63,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790284076800039-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790284076800039-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260924 21:08:07.263688 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAzNNjS/build/release/bin/kudu
/tmp/dist-test-taskAzNNjS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790284076800039-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790284076800039-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790284076800039-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790284076800039-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33355
--keytab_file=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790284076800039-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36479
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790284076800039-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790284076800039-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790284076800039-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 21:08:07.334964 2466 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 21:08:07.335124 2466 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 21:08:07.335145 2466 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 21:08:07.336161 2466 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 21:08:07.336198 2466 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 21:08:07.336216 2466 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260924 21:08:07.336265 2466 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 21:08:07.336289 2466 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 21:08:07.337641 2466 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36479
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790284076800039-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790284076800039-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790284076800039-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790284076800039-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:33355
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2466
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790284076800039-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 19:49:06 UTC on ccycloud.quasar-vhqjoc.root.comops.site
build id 4670864
I20260924 21:08:07.337877 2466 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 21:08:07.338110 2466 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 21:08:07.342737 2471 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 21:08:07.342834 2474 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 21:08:07.342746 2472 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 21:08:07.343091 2466 server_base.cc:1034] running on GCE node
I20260924 21:08:07.343274 2466 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 21:08:07.343539 2466 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 21:08:07.344702 2466 hybrid_clock.cc:648] HybridClock initialized: now 1790284087344691 us; error 50 us; skew 500 ppm
Sep 24 21:08:07 dist-test-slave-c7tk krb5kdc[2069](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284087, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 21:08:07.347010 2466 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 21:08:07.347434 2466 webserver.cc:460] Webserver started at http://127.0.98.3:41459/ using document root <none> and password file <none>
I20260924 21:08:07.347661 2466 fs_manager.cc:359] Metadata directory not provided
I20260924 21:08:07.347703 2466 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 21:08:07.347839 2466 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 21:08:07.348796 2466 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790284076800039-392-0/minicluster-data/ts-2/data/instance:
uuid: "592523081f024e1a976a2d02808f79b4"
format_stamp: "Formatted at 2026-09-24 21:08:07 on dist-test-slave-c7tk"
server_key: "0fad2b85f3e656d40d69a90cfa069c15"
server_key_iv: "0d3f1a195ca71b0b7d5ce5bf6ef4d5e1"
server_key_version: "encryptionkey@0"
I20260924 21:08:07.349123 2466 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790284076800039-392-0/minicluster-data/ts-2/wal/instance:
uuid: "592523081f024e1a976a2d02808f79b4"
format_stamp: "Formatted at 2026-09-24 21:08:07 on dist-test-slave-c7tk"
server_key: "0fad2b85f3e656d40d69a90cfa069c15"
server_key_iv: "0d3f1a195ca71b0b7d5ce5bf6ef4d5e1"
server_key_version: "encryptionkey@0"
I20260924 21:08:07.350390 2466 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 21:08:07.351083 2481 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 21:08:07.351250 2466 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 21:08:07.351356 2466 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790284076800039-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790284076800039-392-0/minicluster-data/ts-2/wal
uuid: "592523081f024e1a976a2d02808f79b4"
format_stamp: "Formatted at 2026-09-24 21:08:07 on dist-test-slave-c7tk"
server_key: "0fad2b85f3e656d40d69a90cfa069c15"
server_key_iv: "0d3f1a195ca71b0b7d5ce5bf6ef4d5e1"
server_key_version: "encryptionkey@0"
I20260924 21:08:07.351438 2466 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790284076800039-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790284076800039-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790284076800039-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 21:08:07.371191 2466 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 21:08:07.372992 2466 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 21:08:07.373128 2466 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 21:08:07.373697 2466 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 21:08:07.373767 2466 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 21:08:07.373872 2466 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 21:08:07.373905 2466 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 21:08:07 dist-test-slave-c7tk krb5kdc[2069](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284087, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 21:08:07.381428 2466 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:33863
I20260924 21:08:07.381450 2596 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:33863 every 8 connection(s)
I20260924 21:08:07.381784 2466 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790284076800039-392-0/minicluster-data/ts-2/data/info.pb
I20260924 21:08:07.384496 2164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:07.374209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:37697 (local address 127.0.98.62:33355)
0924 21:08:07.374251 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0924 21:08:07.374253 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 21:08:07.374519 (+ 266us) server_negotiation.cc:407] Connection header received
0924 21:08:07.375217 (+ 698us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:08:07.375218 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:08:07.375232 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:08:07.375293 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:08:07.376058 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:07.376729 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:07.378545 (+ 1816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:07.378831 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:07.378918 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:07.381622 (+ 2704us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 21:08:07.381624 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 21:08:07.381624 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 21:08:07.381632 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 21:08:07.383179 (+ 1547us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:08:07.383562 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:08:07.383563 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:08:07.383564 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:08:07.383620 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:08:07.383869 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:08:07.383870 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:08:07.383871 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:08:07.383983 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 21:08:07.384062 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0924 21:08:07.384318 (+ 256us) server_negotiation.cc:299] Negotiation successful
0924 21:08:07.384412 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260924 21:08:07.384593 2490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:07.374149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33355 (local address 127.0.98.3:37697)
0924 21:08:07.374366 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0924 21:08:07.374404 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0924 21:08:07.375109 (+ 705us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 21:08:07.375353 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 21:08:07.375356 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 21:08:07.375688 (+ 332us) client_negotiation.cc:189] Negotiated authn=SASL
0924 21:08:07.375936 (+ 248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:07.375943 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:07.376916 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:07.376917 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:07.378400 (+ 1483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:07.378402 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:07.378923 (+ 521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:07.378924 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:07.379203 (+ 279us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:07.379859 (+ 656us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 21:08:07.379869 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 21:08:07.381517 (+ 1648us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 21:08:07.383268 (+ 1751us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 21:08:07.383273 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 21:08:07.383275 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 21:08:07.383476 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 21:08:07.383688 (+ 212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 21:08:07.383689 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 21:08:07.383689 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 21:08:07.383790 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 21:08:07.384052 (+ 262us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 21:08:07.384058 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 21:08:07.384183 (+ 125us) client_negotiation.cc:769] Sending connection context
0924 21:08:07.384283 (+ 100us) client_negotiation.cc:240] Negotiation successful
0924 21:08:07.384414 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":91,"threads_started":1}
I20260924 21:08:07.384948 2597 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33355
I20260924 21:08:07.385023 2597 heartbeater.cc:464] Registering TS with master...
I20260924 21:08:07.385174 2597 heartbeater.cc:511] Master 127.0.98.62:33355 requested a full tablet report, sending...
I20260924 21:08:07.385469 2114 ts_manager.cc:194] Registered new tserver with Master: 592523081f024e1a976a2d02808f79b4 (127.0.98.3:33863)
I20260924 21:08:07.385898 2114 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:37697
I20260924 21:08:07.387513 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAzNNjS/build/release/bin/kudu as pid 2466
I20260924 21:08:07.387624 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790284076800039-392-0/minicluster-data/ts-2/wal/instance
I20260924 21:08:07.387822 392 external_mini_cluster.cc:1428] Setting key 258701afd9cc7cfe27438326d02cb63f
I20260924 21:08:07.388707 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 21:08:07.391394 2164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:07.385889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:33047 (local address 127.0.98.62:33355)
0924 21:08:07.385957 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0924 21:08:07.385959 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 21:08:07.386144 (+ 185us) server_negotiation.cc:407] Connection header received
0924 21:08:07.386201 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:08:07.386202 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:08:07.386211 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:08:07.386253 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:08:07.386514 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:07.386970 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:07.387640 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:07.387871 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:07.387950 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:07.388935 (+ 985us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 21:08:07.388936 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 21:08:07.388937 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 21:08:07.388942 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 21:08:07.390273 (+ 1331us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:08:07.390650 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:08:07.390651 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:08:07.390652 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:08:07.390705 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:08:07.390905 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:08:07.390905 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:08:07.390906 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:08:07.391010 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 21:08:07.391083 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0924 21:08:07.391239 (+ 156us) server_negotiation.cc:299] Negotiation successful
0924 21:08:07.391304 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260924 21:08:07.391400 2603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:07.385852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33355 (local address 127.0.98.3:33047)
0924 21:08:07.386083 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0924 21:08:07.386090 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0924 21:08:07.386153 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 21:08:07.386255 (+ 102us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 21:08:07.386256 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 21:08:07.386400 (+ 144us) client_negotiation.cc:189] Negotiated authn=SASL
0924 21:08:07.386447 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:07.386448 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:07.387060 (+ 612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:07.387061 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:07.387562 (+ 501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:07.387563 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:07.387953 (+ 390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:07.387954 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:07.388040 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:07.388487 (+ 447us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 21:08:07.388492 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 21:08:07.388864 (+ 372us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 21:08:07.390356 (+ 1492us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 21:08:07.390356 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 21:08:07.390357 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 21:08:07.390555 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 21:08:07.390764 (+ 209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 21:08:07.390765 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 21:08:07.390765 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 21:08:07.390817 (+ 52us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 21:08:07.391076 (+ 259us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 21:08:07.391076 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 21:08:07.391117 (+ 41us) client_negotiation.cc:769] Sending connection context
0924 21:08:07.391198 (+ 81us) client_negotiation.cc:240] Negotiation successful
0924 21:08:07.391299 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":117,"threads_started":1}
Sep 24 21:08:07 dist-test-slave-c7tk krb5kdc[2069](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284087, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 24 21:08:07 dist-test-slave-c7tk krb5kdc[2069](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284087, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 21:08:07.409655 2164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:07.403396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57948 (local address 127.0.98.62:33355)
0924 21:08:07.403439 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0924 21:08:07.403442 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 21:08:07.403468 (+ 26us) server_negotiation.cc:407] Connection header received
0924 21:08:07.403536 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:08:07.403536 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:08:07.403545 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:08:07.403604 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:08:07.404083 (+ 479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:07.404565 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:07.405236 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:07.405395 (+ 159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:07.405466 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:07.407570 (+ 2104us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 21:08:07.407571 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 21:08:07.407571 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 21:08:07.407576 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 21:08:07.408813 (+ 1237us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:08:07.409184 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:08:07.409185 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:08:07.409185 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:08:07.409226 (+ 41us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:08:07.409362 (+ 136us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:08:07.409363 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:08:07.409363 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:08:07.409436 (+ 73us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 21:08:07.409499 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0924 21:08:07.409566 (+ 67us) server_negotiation.cc:299] Negotiation successful
0924 21:08:07.409610 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260924 21:08:07.410923 2114 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57948:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 21:08:07.412268 2114 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 21:08:07.417925 2612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:07.414016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42367 (local address 127.0.0.1:45502)
0924 21:08:07.414271 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0924 21:08:07.414278 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0924 21:08:07.414307 (+ 29us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 21:08:07.414636 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 21:08:07.414637 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 21:08:07.414639 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 21:08:07.414707 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:07.414709 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:07.415911 (+ 1202us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:07.415911 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:07.416708 (+ 797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:07.416710 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:07.417658 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:07.417659 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:07.417709 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:07.417719 (+ 10us) client_negotiation.cc:769] Sending connection context
0924 21:08:07.417772 (+ 53us) client_negotiation.cc:240] Negotiation successful
0924 21:08:07.417855 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":248,"mutex_wait_us":21,"thread_start_us":44,"threads_started":1}
I20260924 21:08:07.417941 2616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:07.414308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45502 (local address 127.0.98.2:42367)
0924 21:08:07.414501 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0924 21:08:07.414504 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 21:08:07.414512 (+ 8us) server_negotiation.cc:407] Connection header received
0924 21:08:07.414542 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:08:07.414543 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:08:07.414554 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:08:07.414638 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 21:08:07.414790 (+ 152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:07.415829 (+ 1039us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:07.416798 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:07.417576 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:07.417654 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:07.417706 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0924 21:08:07.417784 (+ 78us) server_negotiation.cc:299] Negotiation successful
0924 21:08:07.417852 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":155,"thread_start_us":99,"threads_started":1}
I20260924 21:08:07.418128 2611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:07.414011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33863 (local address 127.0.0.1:53752)
0924 21:08:07.414137 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0924 21:08:07.414144 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0924 21:08:07.414194 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 21:08:07.414627 (+ 433us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 21:08:07.414630 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 21:08:07.414632 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 21:08:07.414689 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:07.414691 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:07.416035 (+ 1344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:07.416035 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:07.416994 (+ 959us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:07.416995 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:07.417869 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:07.417870 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:07.417940 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:07.417944 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 21:08:07.417999 (+ 55us) client_negotiation.cc:240] Negotiation successful
0924 21:08:07.418051 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":46,"threads_started":1}
I20260924 21:08:07.418352 2615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:07.414174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53752 (local address 127.0.98.3:33863)
0924 21:08:07.414479 (+ 305us) server_negotiation.cc:206] Beginning negotiation
0924 21:08:07.414482 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 21:08:07.414490 (+ 8us) server_negotiation.cc:407] Connection header received
0924 21:08:07.414515 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:08:07.414517 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:08:07.414528 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:08:07.414624 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 21:08:07.414756 (+ 132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:07.415798 (+ 1042us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:07.417074 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:07.417754 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:07.417867 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:07.417918 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0924 21:08:07.418263 (+ 345us) server_negotiation.cc:299] Negotiation successful
0924 21:08:07.418299 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":282,"thread_start_us":59,"threads_started":1}
I20260924 21:08:07.418529 2391 tablet_service.cc:1467] Processing CreateTablet for tablet 2406f074aab3424782d86d6badbcca51 (DEFAULT_TABLE table=test-table [id=1fb1a04f719b48b99bf24be0738784aa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 21:08:07.418823 2391 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2406f074aab3424782d86d6badbcca51. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 21:08:07.418848 2531 tablet_service.cc:1467] Processing CreateTablet for tablet 2406f074aab3424782d86d6badbcca51 (DEFAULT_TABLE table=test-table [id=1fb1a04f719b48b99bf24be0738784aa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 21:08:07.419157 2531 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2406f074aab3424782d86d6badbcca51. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 21:08:07.419395 2613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:07.413961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44497 (local address 127.0.0.1:43754)
0924 21:08:07.414100 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0924 21:08:07.414134 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0924 21:08:07.414194 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 21:08:07.414993 (+ 799us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 21:08:07.414994 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 21:08:07.414996 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 21:08:07.415035 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:07.415036 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:07.415879 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:07.415880 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:07.416991 (+ 1111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:07.416994 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:07.418966 (+ 1972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:07.418967 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:07.419180 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:07.419184 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 21:08:07.419284 (+ 100us) client_negotiation.cc:240] Negotiation successful
0924 21:08:07.419318 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"mutex_wait_us":65,"thread_start_us":36,"threads_started":1}
I20260924 21:08:07.419432 2614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:07.414153 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43754 (local address 127.0.98.1:44497)
0924 21:08:07.414725 (+ 572us) server_negotiation.cc:206] Beginning negotiation
0924 21:08:07.414729 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 21:08:07.414738 (+ 9us) server_negotiation.cc:407] Connection header received
0924 21:08:07.414773 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:08:07.414775 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:08:07.414789 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:08:07.415108 (+ 319us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 21:08:07.415163 (+ 55us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:07.415801 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:07.417953 (+ 2152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:07.418866 (+ 913us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:07.419035 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:07.419088 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0924 21:08:07.419300 (+ 212us) server_negotiation.cc:299] Negotiation successful
0924 21:08:07.419336 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":538,"thread_start_us":65,"threads_started":1}
I20260924 21:08:07.420151 2251 tablet_service.cc:1467] Processing CreateTablet for tablet 2406f074aab3424782d86d6badbcca51 (DEFAULT_TABLE table=test-table [id=1fb1a04f719b48b99bf24be0738784aa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 21:08:07.420449 2251 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2406f074aab3424782d86d6badbcca51. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 21:08:07.421536 2617 tablet_bootstrap.cc:492] T 2406f074aab3424782d86d6badbcca51 P 7a5f5a1ce0964d918460512bd4a094d1: Bootstrap starting.
I20260924 21:08:07.421793 2618 tablet_bootstrap.cc:492] T 2406f074aab3424782d86d6badbcca51 P 592523081f024e1a976a2d02808f79b4: Bootstrap starting.
I20260924 21:08:07.422286 2617 tablet_bootstrap.cc:654] T 2406f074aab3424782d86d6badbcca51 P 7a5f5a1ce0964d918460512bd4a094d1: Neither blocks nor log segments found. Creating new log.
I20260924 21:08:07.422592 2618 tablet_bootstrap.cc:654] T 2406f074aab3424782d86d6badbcca51 P 592523081f024e1a976a2d02808f79b4: Neither blocks nor log segments found. Creating new log.
I20260924 21:08:07.422580 2617 log.cc:824] T 2406f074aab3424782d86d6badbcca51 P 7a5f5a1ce0964d918460512bd4a094d1: Log is configured to *not* fsync() on all Append() calls
I20260924 21:08:07.422873 2618 log.cc:824] T 2406f074aab3424782d86d6badbcca51 P 592523081f024e1a976a2d02808f79b4: Log is configured to *not* fsync() on all Append() calls
I20260924 21:08:07.423285 2617 tablet_bootstrap.cc:492] T 2406f074aab3424782d86d6badbcca51 P 7a5f5a1ce0964d918460512bd4a094d1: No bootstrap required, opened a new log
I20260924 21:08:07.423383 2621 tablet_bootstrap.cc:492] T 2406f074aab3424782d86d6badbcca51 P 508ed3a686554bd6af0204840904ab09: Bootstrap starting.
I20260924 21:08:07.423465 2618 tablet_bootstrap.cc:492] T 2406f074aab3424782d86d6badbcca51 P 592523081f024e1a976a2d02808f79b4: No bootstrap required, opened a new log
I20260924 21:08:07.423435 2617 ts_tablet_manager.cc:1397] T 2406f074aab3424782d86d6badbcca51 P 7a5f5a1ce0964d918460512bd4a094d1: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260924 21:08:07.423518 2618 ts_tablet_manager.cc:1397] T 2406f074aab3424782d86d6badbcca51 P 592523081f024e1a976a2d02808f79b4: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260924 21:08:07.423934 2621 tablet_bootstrap.cc:654] T 2406f074aab3424782d86d6badbcca51 P 508ed3a686554bd6af0204840904ab09: Neither blocks nor log segments found. Creating new log.
I20260924 21:08:07.424197 2621 log.cc:824] T 2406f074aab3424782d86d6badbcca51 P 508ed3a686554bd6af0204840904ab09: Log is configured to *not* fsync() on all Append() calls
I20260924 21:08:07.424749 2621 tablet_bootstrap.cc:492] T 2406f074aab3424782d86d6badbcca51 P 508ed3a686554bd6af0204840904ab09: No bootstrap required, opened a new log
I20260924 21:08:07.424817 2621 ts_tablet_manager.cc:1397] T 2406f074aab3424782d86d6badbcca51 P 508ed3a686554bd6af0204840904ab09: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 21:08:07.425148 2618 raft_consensus.cc:348] T 2406f074aab3424782d86d6badbcca51 P 592523081f024e1a976a2d02808f79b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "508ed3a686554bd6af0204840904ab09" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44497 } } peers { permanent_uuid: "592523081f024e1a976a2d02808f79b4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33863 } } peers { permanent_uuid: "7a5f5a1ce0964d918460512bd4a094d1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42367 } }
I20260924 21:08:07.425148 2617 raft_consensus.cc:348] T 2406f074aab3424782d86d6badbcca51 P 7a5f5a1ce0964d918460512bd4a094d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "508ed3a686554bd6af0204840904ab09" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44497 } } peers { permanent_uuid: "592523081f024e1a976a2d02808f79b4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33863 } } peers { permanent_uuid: "7a5f5a1ce0964d918460512bd4a094d1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42367 } }
I20260924 21:08:07.425287 2617 raft_consensus.cc:374] T 2406f074aab3424782d86d6badbcca51 P 7a5f5a1ce0964d918460512bd4a094d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 21:08:07.425287 2618 raft_consensus.cc:374] T 2406f074aab3424782d86d6badbcca51 P 592523081f024e1a976a2d02808f79b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 21:08:07.425318 2617 raft_consensus.cc:729] T 2406f074aab3424782d86d6badbcca51 P 7a5f5a1ce0964d918460512bd4a094d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a5f5a1ce0964d918460512bd4a094d1, State: Initialized, Role: FOLLOWER
I20260924 21:08:07.425318 2618 raft_consensus.cc:729] T 2406f074aab3424782d86d6badbcca51 P 592523081f024e1a976a2d02808f79b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 592523081f024e1a976a2d02808f79b4, State: Initialized, Role: FOLLOWER
I20260924 21:08:07.425414 2618 consensus_queue.cc:260] T 2406f074aab3424782d86d6badbcca51 P 592523081f024e1a976a2d02808f79b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "508ed3a686554bd6af0204840904ab09" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44497 } } peers { permanent_uuid: "592523081f024e1a976a2d02808f79b4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33863 } } peers { permanent_uuid: "7a5f5a1ce0964d918460512bd4a094d1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42367 } }
I20260924 21:08:07.425432 2617 consensus_queue.cc:260] T 2406f074aab3424782d86d6badbcca51 P 7a5f5a1ce0964d918460512bd4a094d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "508ed3a686554bd6af0204840904ab09" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44497 } } peers { permanent_uuid: "592523081f024e1a976a2d02808f79b4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33863 } } peers { permanent_uuid: "7a5f5a1ce0964d918460512bd4a094d1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42367 } }
I20260924 21:08:07.425637 2618 ts_tablet_manager.cc:1428] T 2406f074aab3424782d86d6badbcca51 P 592523081f024e1a976a2d02808f79b4: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260924 21:08:07.425637 2617 ts_tablet_manager.cc:1428] T 2406f074aab3424782d86d6badbcca51 P 7a5f5a1ce0964d918460512bd4a094d1: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260924 21:08:07.425737 2597 heartbeater.cc:503] Master 127.0.98.62:33355 was elected leader, sending a full tablet report...
I20260924 21:08:07.425768 2457 heartbeater.cc:503] Master 127.0.98.62:33355 was elected leader, sending a full tablet report...
I20260924 21:08:07.426352 2621 raft_consensus.cc:348] T 2406f074aab3424782d86d6badbcca51 P 508ed3a686554bd6af0204840904ab09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "508ed3a686554bd6af0204840904ab09" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44497 } } peers { permanent_uuid: "592523081f024e1a976a2d02808f79b4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33863 } } peers { permanent_uuid: "7a5f5a1ce0964d918460512bd4a094d1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42367 } }
I20260924 21:08:07.426491 2621 raft_consensus.cc:374] T 2406f074aab3424782d86d6badbcca51 P 508ed3a686554bd6af0204840904ab09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 21:08:07.426525 2621 raft_consensus.cc:729] T 2406f074aab3424782d86d6badbcca51 P 508ed3a686554bd6af0204840904ab09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 508ed3a686554bd6af0204840904ab09, State: Initialized, Role: FOLLOWER
I20260924 21:08:07.426617 2621 consensus_queue.cc:260] T 2406f074aab3424782d86d6badbcca51 P 508ed3a686554bd6af0204840904ab09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "508ed3a686554bd6af0204840904ab09" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44497 } } peers { permanent_uuid: "592523081f024e1a976a2d02808f79b4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33863 } } peers { permanent_uuid: "7a5f5a1ce0964d918460512bd4a094d1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42367 } }
I20260924 21:08:07.426872 2621 ts_tablet_manager.cc:1428] T 2406f074aab3424782d86d6badbcca51 P 508ed3a686554bd6af0204840904ab09: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 21:08:07.426915 2317 heartbeater.cc:503] Master 127.0.98.62:33355 was elected leader, sending a full tablet report...
I20260924 21:08:07.458797 2624 raft_consensus.cc:482] T 2406f074aab3424782d86d6badbcca51 P 592523081f024e1a976a2d02808f79b4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 21:08:07.458854 2624 raft_consensus.cc:504] T 2406f074aab3424782d86d6badbcca51 P 592523081f024e1a976a2d02808f79b4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "508ed3a686554bd6af0204840904ab09" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44497 } } peers { permanent_uuid: "592523081f024e1a976a2d02808f79b4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33863 } } peers { permanent_uuid: "7a5f5a1ce0964d918460512bd4a094d1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42367 } }
I20260924 21:08:07.459134 2624 leader_election.cc:288] T 2406f074aab3424782d86d6badbcca51 P 592523081f024e1a976a2d02808f79b4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 508ed3a686554bd6af0204840904ab09 (127.0.98.1:44497), 7a5f5a1ce0964d918460512bd4a094d1 (127.0.98.2:42367)
I20260924 21:08:07.461977 2614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:07.459374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:43855 (local address 127.0.98.1:44497)
0924 21:08:07.459422 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0924 21:08:07.459424 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 21:08:07.459430 (+ 6us) server_negotiation.cc:407] Connection header received
0924 21:08:07.459452 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:08:07.459453 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:08:07.459460 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:08:07.459551 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 21:08:07.459691 (+ 140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:07.460342 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:07.461219 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:07.461586 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:07.461637 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:07.461642 (+ 5us) server_negotiation.cc:1035] Waiting for connection context
0924 21:08:07.461881 (+ 239us) server_negotiation.cc:299] Negotiation successful
0924 21:08:07.461923 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260924 21:08:07.462013 2616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:07.459365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:39307 (local address 127.0.98.2:42367)
0924 21:08:07.459400 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0924 21:08:07.459401 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 21:08:07.459555 (+ 154us) server_negotiation.cc:407] Connection header received
0924 21:08:07.459577 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:08:07.459578 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:08:07.459586 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:08:07.459627 (+ 41us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 21:08:07.459755 (+ 128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:07.460358 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:07.461261 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:07.461661 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:07.461738 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:07.461747 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0924 21:08:07.461921 (+ 174us) server_negotiation.cc:299] Negotiation successful
0924 21:08:07.461955 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260924 21:08:07.461992 2490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:07.459309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44497 (local address 127.0.98.3:43855)
0924 21:08:07.459354 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0924 21:08:07.459364 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 21:08:07.459404 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 21:08:07.459560 (+ 156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 21:08:07.459561 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 21:08:07.459562 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 21:08:07.459615 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:07.459616 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:07.460416 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:07.460417 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:07.461135 (+ 718us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:07.461137 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:07.461673 (+ 536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:07.461674 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:07.461756 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:07.461760 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 21:08:07.461876 (+ 116us) client_negotiation.cc:240] Negotiation successful
0924 21:08:07.461918 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31,"mutex_wait_us":24}
I20260924 21:08:07.462221 2626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:07.459318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42367 (local address 127.0.98.3:39307)
0924 21:08:07.459466 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0924 21:08:07.459475 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 21:08:07.459530 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 21:08:07.459632 (+ 102us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 21:08:07.459632 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 21:08:07.459634 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 21:08:07.459698 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:07.459699 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:07.460439 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:07.460441 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:07.461150 (+ 709us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:07.461152 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:07.461742 (+ 590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:07.461742 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:07.461846 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:07.461851 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 21:08:07.461915 (+ 64us) client_negotiation.cc:240] Negotiation successful
0924 21:08:07.461946 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"mutex_wait_us":10,"thread_start_us":88,"threads_started":1}
I20260924 21:08:07.462296 2271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2406f074aab3424782d86d6badbcca51" candidate_uuid: "592523081f024e1a976a2d02808f79b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "508ed3a686554bd6af0204840904ab09" is_pre_election: true
I20260924 21:08:07.462394 2271 raft_consensus.cc:2454] T 2406f074aab3424782d86d6badbcca51 P 508ed3a686554bd6af0204840904ab09 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 592523081f024e1a976a2d02808f79b4 in term 0.
I20260924 21:08:07.462565 2485 leader_election.cc:302] T 2406f074aab3424782d86d6badbcca51 P 592523081f024e1a976a2d02808f79b4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 508ed3a686554bd6af0204840904ab09, 592523081f024e1a976a2d02808f79b4; no voters:
I20260924 21:08:07.462559 2411 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2406f074aab3424782d86d6badbcca51" candidate_uuid: "592523081f024e1a976a2d02808f79b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a5f5a1ce0964d918460512bd4a094d1" is_pre_election: true
I20260924 21:08:07.462689 2411 raft_consensus.cc:2454] T 2406f074aab3424782d86d6badbcca51 P 7a5f5a1ce0964d918460512bd4a094d1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 592523081f024e1a976a2d02808f79b4 in term 0.
I20260924 21:08:07.462738 2624 raft_consensus.cc:2781] T 2406f074aab3424782d86d6badbcca51 P 592523081f024e1a976a2d02808f79b4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 21:08:07.462803 2624 raft_consensus.cc:482] T 2406f074aab3424782d86d6badbcca51 P 592523081f024e1a976a2d02808f79b4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 21:08:07.462831 2624 raft_consensus.cc:3037] T 2406f074aab3424782d86d6badbcca51 P 592523081f024e1a976a2d02808f79b4 [term 0 FOLLOWER]: Advancing to term 1
I20260924 21:08:07.463379 2624 raft_consensus.cc:504] T 2406f074aab3424782d86d6badbcca51 P 592523081f024e1a976a2d02808f79b4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "508ed3a686554bd6af0204840904ab09" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44497 } } peers { permanent_uuid: "592523081f024e1a976a2d02808f79b4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33863 } } peers { permanent_uuid: "7a5f5a1ce0964d918460512bd4a094d1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42367 } }
I20260924 21:08:07.463511 2624 leader_election.cc:288] T 2406f074aab3424782d86d6badbcca51 P 592523081f024e1a976a2d02808f79b4 [CANDIDATE]: Term 1 election: Requested vote from peers 508ed3a686554bd6af0204840904ab09 (127.0.98.1:44497), 7a5f5a1ce0964d918460512bd4a094d1 (127.0.98.2:42367)
I20260924 21:08:07.463661 2271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2406f074aab3424782d86d6badbcca51" candidate_uuid: "592523081f024e1a976a2d02808f79b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "508ed3a686554bd6af0204840904ab09"
I20260924 21:08:07.463764 2271 raft_consensus.cc:3037] T 2406f074aab3424782d86d6badbcca51 P 508ed3a686554bd6af0204840904ab09 [term 0 FOLLOWER]: Advancing to term 1
I20260924 21:08:07.464427 2271 raft_consensus.cc:2454] T 2406f074aab3424782d86d6badbcca51 P 508ed3a686554bd6af0204840904ab09 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 592523081f024e1a976a2d02808f79b4 in term 1.
I20260924 21:08:07.464552 2411 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2406f074aab3424782d86d6badbcca51" candidate_uuid: "592523081f024e1a976a2d02808f79b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a5f5a1ce0964d918460512bd4a094d1"
I20260924 21:08:07.464586 2485 leader_election.cc:302] T 2406f074aab3424782d86d6badbcca51 P 592523081f024e1a976a2d02808f79b4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 508ed3a686554bd6af0204840904ab09, 592523081f024e1a976a2d02808f79b4; no voters:
I20260924 21:08:07.464633 2411 raft_consensus.cc:3037] T 2406f074aab3424782d86d6badbcca51 P 7a5f5a1ce0964d918460512bd4a094d1 [term 0 FOLLOWER]: Advancing to term 1
I20260924 21:08:07.464655 2624 raft_consensus.cc:2781] T 2406f074aab3424782d86d6badbcca51 P 592523081f024e1a976a2d02808f79b4 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 21:08:07.464771 2624 raft_consensus.cc:686] T 2406f074aab3424782d86d6badbcca51 P 592523081f024e1a976a2d02808f79b4 [term 1 LEADER]: Becoming Leader. State: Replica: 592523081f024e1a976a2d02808f79b4, State: Running, Role: LEADER
I20260924 21:08:07.464852 2624 consensus_queue.cc:237] T 2406f074aab3424782d86d6badbcca51 P 592523081f024e1a976a2d02808f79b4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "508ed3a686554bd6af0204840904ab09" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44497 } } peers { permanent_uuid: "592523081f024e1a976a2d02808f79b4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33863 } } peers { permanent_uuid: "7a5f5a1ce0964d918460512bd4a094d1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42367 } }
I20260924 21:08:07.465483 2411 raft_consensus.cc:2454] T 2406f074aab3424782d86d6badbcca51 P 7a5f5a1ce0964d918460512bd4a094d1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 592523081f024e1a976a2d02808f79b4 in term 1.
I20260924 21:08:07.465607 2114 catalog_manager.cc:5495] T 2406f074aab3424782d86d6badbcca51 P 592523081f024e1a976a2d02808f79b4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 592523081f024e1a976a2d02808f79b4 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "592523081f024e1a976a2d02808f79b4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "508ed3a686554bd6af0204840904ab09" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44497 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "592523081f024e1a976a2d02808f79b4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33863 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7a5f5a1ce0964d918460512bd4a094d1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42367 } health_report { overall_health: UNKNOWN } } }
W20260924 21:08:07.490845 2458 tablet.cc:2369] T 2406f074aab3424782d86d6badbcca51 P 7a5f5a1ce0964d918460512bd4a094d1: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 24 21:08:07 dist-test-slave-c7tk krb5kdc[2069](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284086, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 21:08:07.507707 2634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:07.502240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33355 (local address 127.0.0.1:57954)
0924 21:08:07.502348 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0924 21:08:07.502355 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0924 21:08:07.502414 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 21:08:07.502505 (+ 91us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 21:08:07.502506 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 21:08:07.502723 (+ 217us) client_negotiation.cc:189] Negotiated authn=SASL
0924 21:08:07.502753 (+ 30us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:07.502755 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:07.503299 (+ 544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:07.503300 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:07.503772 (+ 472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:07.503773 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:07.504017 (+ 244us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:07.504018 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:07.504050 (+ 32us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:07.504411 (+ 361us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 21:08:07.504415 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 21:08:07.505573 (+ 1158us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 21:08:07.506848 (+ 1275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 21:08:07.506871 (+ 23us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 21:08:07.506873 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 21:08:07.507043 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 21:08:07.507208 (+ 165us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 21:08:07.507209 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 21:08:07.507209 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 21:08:07.507262 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 21:08:07.507464 (+ 202us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 21:08:07.507469 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 21:08:07.507506 (+ 37us) client_negotiation.cc:769] Sending connection context
0924 21:08:07.507560 (+ 54us) client_negotiation.cc:240] Negotiation successful
0924 21:08:07.507628 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":81,"thread_start_us":54,"threads_started":1}
I20260924 21:08:07.507766 2164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:07.502298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57954 (local address 127.0.98.62:33355)
0924 21:08:07.502352 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0924 21:08:07.502354 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 21:08:07.502403 (+ 49us) server_negotiation.cc:407] Connection header received
0924 21:08:07.502446 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:08:07.502446 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:08:07.502453 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:08:07.502493 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:08:07.502807 (+ 314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:07.503248 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:07.503823 (+ 575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:07.503966 (+ 143us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:07.504007 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:07.505642 (+ 1635us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 21:08:07.505643 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 21:08:07.505643 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 21:08:07.505648 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 21:08:07.506786 (+ 1138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:08:07.507105 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:08:07.507106 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:08:07.507106 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:08:07.507153 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:08:07.507312 (+ 159us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:08:07.507313 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:08:07.507313 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:08:07.507414 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 21:08:07.507461 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0924 21:08:07.507605 (+ 144us) server_negotiation.cc:299] Negotiation successful
0924 21:08:07.507668 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260924 21:08:07.509425 2114 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:57954:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260924 21:08:07.509575 2114 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 21:08:07.510691 2391 tablet_service.cc:1467] Processing CreateTablet for tablet ec9a7eb6eb89431dae5e4e8bf3f597f3 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=4b1cb6f789474b35b2c3c76d2c78e80a]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 21:08:07.510730 2251 tablet_service.cc:1467] Processing CreateTablet for tablet ec9a7eb6eb89431dae5e4e8bf3f597f3 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=4b1cb6f789474b35b2c3c76d2c78e80a]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 21:08:07.510725 2531 tablet_service.cc:1467] Processing CreateTablet for tablet ec9a7eb6eb89431dae5e4e8bf3f597f3 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=4b1cb6f789474b35b2c3c76d2c78e80a]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 21:08:07.510831 2391 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ec9a7eb6eb89431dae5e4e8bf3f597f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 21:08:07.510838 2531 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ec9a7eb6eb89431dae5e4e8bf3f597f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 21:08:07.510839 2251 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ec9a7eb6eb89431dae5e4e8bf3f597f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 21:08:07.511684 2617 tablet_bootstrap.cc:492] T ec9a7eb6eb89431dae5e4e8bf3f597f3 P 7a5f5a1ce0964d918460512bd4a094d1: Bootstrap starting.
I20260924 21:08:07.512097 2621 tablet_bootstrap.cc:492] T ec9a7eb6eb89431dae5e4e8bf3f597f3 P 508ed3a686554bd6af0204840904ab09: Bootstrap starting.
I20260924 21:08:07.512188 2617 tablet_bootstrap.cc:654] T ec9a7eb6eb89431dae5e4e8bf3f597f3 P 7a5f5a1ce0964d918460512bd4a094d1: Neither blocks nor log segments found. Creating new log.
I20260924 21:08:07.512418 2618 tablet_bootstrap.cc:492] T ec9a7eb6eb89431dae5e4e8bf3f597f3 P 592523081f024e1a976a2d02808f79b4: Bootstrap starting.
I20260924 21:08:07.512688 2621 tablet_bootstrap.cc:654] T ec9a7eb6eb89431dae5e4e8bf3f597f3 P 508ed3a686554bd6af0204840904ab09: Neither blocks nor log segments found. Creating new log.
I20260924 21:08:07.512735 2617 tablet_bootstrap.cc:492] T ec9a7eb6eb89431dae5e4e8bf3f597f3 P 7a5f5a1ce0964d918460512bd4a094d1: No bootstrap required, opened a new log
I20260924 21:08:07.512751 2618 tablet_bootstrap.cc:654] T ec9a7eb6eb89431dae5e4e8bf3f597f3 P 592523081f024e1a976a2d02808f79b4: Neither blocks nor log segments found. Creating new log.
I20260924 21:08:07.512780 2617 ts_tablet_manager.cc:1397] T ec9a7eb6eb89431dae5e4e8bf3f597f3 P 7a5f5a1ce0964d918460512bd4a094d1: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260924 21:08:07.512930 2617 raft_consensus.cc:348] T ec9a7eb6eb89431dae5e4e8bf3f597f3 P 7a5f5a1ce0964d918460512bd4a094d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "508ed3a686554bd6af0204840904ab09" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44497 } } peers { permanent_uuid: "7a5f5a1ce0964d918460512bd4a094d1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42367 } } peers { permanent_uuid: "592523081f024e1a976a2d02808f79b4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33863 } }
I20260924 21:08:07.513003 2617 raft_consensus.cc:374] T ec9a7eb6eb89431dae5e4e8bf3f597f3 P 7a5f5a1ce0964d918460512bd4a094d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 21:08:07.513023 2617 raft_consensus.cc:729] T ec9a7eb6eb89431dae5e4e8bf3f597f3 P 7a5f5a1ce0964d918460512bd4a094d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a5f5a1ce0964d918460512bd4a094d1, State: Initialized, Role: FOLLOWER
I20260924 21:08:07.513073 2617 consensus_queue.cc:260] T ec9a7eb6eb89431dae5e4e8bf3f597f3 P 7a5f5a1ce0964d918460512bd4a094d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "508ed3a686554bd6af0204840904ab09" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44497 } } peers { permanent_uuid: "7a5f5a1ce0964d918460512bd4a094d1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42367 } } peers { permanent_uuid: "592523081f024e1a976a2d02808f79b4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33863 } }
I20260924 21:08:07.513147 2617 ts_tablet_manager.cc:1428] T ec9a7eb6eb89431dae5e4e8bf3f597f3 P 7a5f5a1ce0964d918460512bd4a094d1: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260924 21:08:07.513262 2623 tablet_replica.cc:439] T ec9a7eb6eb89431dae5e4e8bf3f597f3 P 7a5f5a1ce0964d918460512bd4a094d1: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "508ed3a686554bd6af0204840904ab09" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44497 } } peers { permanent_uuid: "7a5f5a1ce0964d918460512bd4a094d1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42367 } } peers { permanent_uuid: "592523081f024e1a976a2d02808f79b4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33863 } } }
I20260924 21:08:07.513327 2623 tablet_replica.cc:442] T ec9a7eb6eb89431dae5e4e8bf3f597f3 P 7a5f5a1ce0964d918460512bd4a094d1: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 21:08:07.513339 2621 tablet_bootstrap.cc:492] T ec9a7eb6eb89431dae5e4e8bf3f597f3 P 508ed3a686554bd6af0204840904ab09: No bootstrap required, opened a new log
I20260924 21:08:07.513392 2621 ts_tablet_manager.cc:1397] T ec9a7eb6eb89431dae5e4e8bf3f597f3 P 508ed3a686554bd6af0204840904ab09: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260924 21:08:07.513553 2621 raft_consensus.cc:348] T ec9a7eb6eb89431dae5e4e8bf3f597f3 P 508ed3a686554bd6af0204840904ab09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "508ed3a686554bd6af0204840904ab09" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44497 } } peers { permanent_uuid: "7a5f5a1ce0964d918460512bd4a094d1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42367 } } peers { permanent_uuid: "592523081f024e1a976a2d02808f79b4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33863 } }
I20260924 21:08:07.513643 2621 raft_consensus.cc:374] T ec9a7eb6eb89431dae5e4e8bf3f597f3 P 508ed3a686554bd6af0204840904ab09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 21:08:07.513670 2621 raft_consensus.cc:729] T ec9a7eb6eb89431dae5e4e8bf3f597f3 P 508ed3a686554bd6af0204840904ab09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 508ed3a686554bd6af0204840904ab09, State: Initialized, Role: FOLLOWER
I20260924 21:08:07.513711 2621 consensus_queue.cc:260] T ec9a7eb6eb89431dae5e4e8bf3f597f3 P 508ed3a686554bd6af0204840904ab09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "508ed3a686554bd6af0204840904ab09" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44497 } } peers { permanent_uuid: "7a5f5a1ce0964d918460512bd4a094d1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42367 } } peers { permanent_uuid: "592523081f024e1a976a2d02808f79b4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33863 } }
I20260924 21:08:07.513787 2618 tablet_bootstrap.cc:492] T ec9a7eb6eb89431dae5e4e8bf3f597f3 P 592523081f024e1a976a2d02808f79b4: No bootstrap required, opened a new log
I20260924 21:08:07.513790 2621 ts_tablet_manager.cc:1428] T ec9a7eb6eb89431dae5e4e8bf3f597f3 P 508ed3a686554bd6af0204840904ab09: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 21:08:07.513849 2618 ts_tablet_manager.cc:1397] T ec9a7eb6eb89431dae5e4e8bf3f597f3 P 592523081f024e1a976a2d02808f79b4: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260924 21:08:07.513893 2625 tablet_replica.cc:439] T ec9a7eb6eb89431dae5e4e8bf3f597f3 P 508ed3a686554bd6af0204840904ab09: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "508ed3a686554bd6af0204840904ab09" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44497 } } peers { permanent_uuid: "7a5f5a1ce0964d918460512bd4a094d1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42367 } } peers { permanent_uuid: "592523081f024e1a976a2d02808f79b4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33863 } } }
I20260924 21:08:07.514004 2625 tablet_replica.cc:442] T ec9a7eb6eb89431dae5e4e8bf3f597f3 P 508ed3a686554bd6af0204840904ab09: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 21:08:07.513994 2618 raft_consensus.cc:348] T ec9a7eb6eb89431dae5e4e8bf3f597f3 P 592523081f024e1a976a2d02808f79b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "508ed3a686554bd6af0204840904ab09" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44497 } } peers { permanent_uuid: "7a5f5a1ce0964d918460512bd4a094d1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42367 } } peers { permanent_uuid: "592523081f024e1a976a2d02808f79b4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33863 } }
I20260924 21:08:07.514051 2618 raft_consensus.cc:374] T ec9a7eb6eb89431dae5e4e8bf3f597f3 P 592523081f024e1a976a2d02808f79b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 21:08:07.514070 2618 raft_consensus.cc:729] T ec9a7eb6eb89431dae5e4e8bf3f597f3 P 592523081f024e1a976a2d02808f79b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 592523081f024e1a976a2d02808f79b4, State: Initialized, Role: FOLLOWER
I20260924 21:08:07.514115 2618 consensus_queue.cc:260] T ec9a7eb6eb89431dae5e4e8bf3f597f3 P 592523081f024e1a976a2d02808f79b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "508ed3a686554bd6af0204840904ab09" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44497 } } peers { permanent_uuid: "7a5f5a1ce0964d918460512bd4a094d1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42367 } } peers { permanent_uuid: "592523081f024e1a976a2d02808f79b4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33863 } }
I20260924 21:08:07.514196 2618 ts_tablet_manager.cc:1428] T ec9a7eb6eb89431dae5e4e8bf3f597f3 P 592523081f024e1a976a2d02808f79b4: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 21:08:07.514242 2624 tablet_replica.cc:439] T ec9a7eb6eb89431dae5e4e8bf3f597f3 P 592523081f024e1a976a2d02808f79b4: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "508ed3a686554bd6af0204840904ab09" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44497 } } peers { permanent_uuid: "7a5f5a1ce0964d918460512bd4a094d1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42367 } } peers { permanent_uuid: "592523081f024e1a976a2d02808f79b4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33863 } } }
I20260924 21:08:07.514320 2624 tablet_replica.cc:442] T ec9a7eb6eb89431dae5e4e8bf3f597f3 P 592523081f024e1a976a2d02808f79b4: This TxnStatusTablet replica's current role is: FOLLOWER
W20260924 21:08:07.587309 2318 tablet.cc:2369] T 2406f074aab3424782d86d6badbcca51 P 508ed3a686554bd6af0204840904ab09: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 21:08:07.632596 2598 tablet.cc:2369] T 2406f074aab3424782d86d6badbcca51 P 592523081f024e1a976a2d02808f79b4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 21:08:07.674346 2624 raft_consensus.cc:482] T ec9a7eb6eb89431dae5e4e8bf3f597f3 P 592523081f024e1a976a2d02808f79b4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 21:08:07.674467 2624 raft_consensus.cc:504] T ec9a7eb6eb89431dae5e4e8bf3f597f3 P 592523081f024e1a976a2d02808f79b4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "508ed3a686554bd6af0204840904ab09" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44497 } } peers { permanent_uuid: "7a5f5a1ce0964d918460512bd4a094d1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42367 } } peers { permanent_uuid: "592523081f024e1a976a2d02808f79b4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33863 } }
I20260924 21:08:07.674711 2624 leader_election.cc:288] T ec9a7eb6eb89431dae5e4e8bf3f597f3 P 592523081f024e1a976a2d02808f79b4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 508ed3a686554bd6af0204840904ab09 (127.0.98.1:44497), 7a5f5a1ce0964d918460512bd4a094d1 (127.0.98.2:42367)
I20260924 21:08:07.674976 2271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec9a7eb6eb89431dae5e4e8bf3f597f3" candidate_uuid: "592523081f024e1a976a2d02808f79b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "508ed3a686554bd6af0204840904ab09" is_pre_election: true
I20260924 21:08:07.675053 2271 raft_consensus.cc:2454] T ec9a7eb6eb89431dae5e4e8bf3f597f3 P 508ed3a686554bd6af0204840904ab09 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 592523081f024e1a976a2d02808f79b4 in term 0.
I20260924 21:08:07.674997 2411 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec9a7eb6eb89431dae5e4e8bf3f597f3" candidate_uuid: "592523081f024e1a976a2d02808f79b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a5f5a1ce0964d918460512bd4a094d1" is_pre_election: true
I20260924 21:08:07.675104 2411 raft_consensus.cc:2454] T ec9a7eb6eb89431dae5e4e8bf3f597f3 P 7a5f5a1ce0964d918460512bd4a094d1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 592523081f024e1a976a2d02808f79b4 in term 0.
I20260924 21:08:07.675202 2485 leader_election.cc:302] T ec9a7eb6eb89431dae5e4e8bf3f597f3 P 592523081f024e1a976a2d02808f79b4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 508ed3a686554bd6af0204840904ab09, 592523081f024e1a976a2d02808f79b4; no voters:
I20260924 21:08:07.675319 2624 raft_consensus.cc:2781] T ec9a7eb6eb89431dae5e4e8bf3f597f3 P 592523081f024e1a976a2d02808f79b4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 21:08:07.675379 2624 raft_consensus.cc:482] T ec9a7eb6eb89431dae5e4e8bf3f597f3 P 592523081f024e1a976a2d02808f79b4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 21:08:07.675410 2624 raft_consensus.cc:3037] T ec9a7eb6eb89431dae5e4e8bf3f597f3 P 592523081f024e1a976a2d02808f79b4 [term 0 FOLLOWER]: Advancing to term 1
I20260924 21:08:07.676450 2624 raft_consensus.cc:504] T ec9a7eb6eb89431dae5e4e8bf3f597f3 P 592523081f024e1a976a2d02808f79b4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "508ed3a686554bd6af0204840904ab09" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44497 } } peers { permanent_uuid: "7a5f5a1ce0964d918460512bd4a094d1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42367 } } peers { permanent_uuid: "592523081f024e1a976a2d02808f79b4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33863 } }
I20260924 21:08:07.676575 2624 leader_election.cc:288] T ec9a7eb6eb89431dae5e4e8bf3f597f3 P 592523081f024e1a976a2d02808f79b4 [CANDIDATE]: Term 1 election: Requested vote from peers 508ed3a686554bd6af0204840904ab09 (127.0.98.1:44497), 7a5f5a1ce0964d918460512bd4a094d1 (127.0.98.2:42367)
I20260924 21:08:07.676699 2271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec9a7eb6eb89431dae5e4e8bf3f597f3" candidate_uuid: "592523081f024e1a976a2d02808f79b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "508ed3a686554bd6af0204840904ab09"
I20260924 21:08:07.676738 2411 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec9a7eb6eb89431dae5e4e8bf3f597f3" candidate_uuid: "592523081f024e1a976a2d02808f79b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a5f5a1ce0964d918460512bd4a094d1"
I20260924 21:08:07.676775 2271 raft_consensus.cc:3037] T ec9a7eb6eb89431dae5e4e8bf3f597f3 P 508ed3a686554bd6af0204840904ab09 [term 0 FOLLOWER]: Advancing to term 1
I20260924 21:08:07.676795 2411 raft_consensus.cc:3037] T ec9a7eb6eb89431dae5e4e8bf3f597f3 P 7a5f5a1ce0964d918460512bd4a094d1 [term 0 FOLLOWER]: Advancing to term 1
I20260924 21:08:07.677515 2411 raft_consensus.cc:2454] T ec9a7eb6eb89431dae5e4e8bf3f597f3 P 7a5f5a1ce0964d918460512bd4a094d1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 592523081f024e1a976a2d02808f79b4 in term 1.
I20260924 21:08:07.677515 2271 raft_consensus.cc:2454] T ec9a7eb6eb89431dae5e4e8bf3f597f3 P 508ed3a686554bd6af0204840904ab09 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 592523081f024e1a976a2d02808f79b4 in term 1.
I20260924 21:08:07.677695 2485 leader_election.cc:302] T ec9a7eb6eb89431dae5e4e8bf3f597f3 P 592523081f024e1a976a2d02808f79b4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 508ed3a686554bd6af0204840904ab09, 592523081f024e1a976a2d02808f79b4; no voters:
I20260924 21:08:07.677810 2624 raft_consensus.cc:2781] T ec9a7eb6eb89431dae5e4e8bf3f597f3 P 592523081f024e1a976a2d02808f79b4 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 21:08:07.677870 2624 raft_consensus.cc:686] T ec9a7eb6eb89431dae5e4e8bf3f597f3 P 592523081f024e1a976a2d02808f79b4 [term 1 LEADER]: Becoming Leader. State: Replica: 592523081f024e1a976a2d02808f79b4, State: Running, Role: LEADER
I20260924 21:08:07.677907 2624 consensus_queue.cc:237] T ec9a7eb6eb89431dae5e4e8bf3f597f3 P 592523081f024e1a976a2d02808f79b4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "508ed3a686554bd6af0204840904ab09" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44497 } } peers { permanent_uuid: "7a5f5a1ce0964d918460512bd4a094d1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42367 } } peers { permanent_uuid: "592523081f024e1a976a2d02808f79b4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33863 } }
I20260924 21:08:07.678339 2627 tablet_replica.cc:439] T ec9a7eb6eb89431dae5e4e8bf3f597f3 P 592523081f024e1a976a2d02808f79b4: TxnStatusTablet state changed. Reason: New leader 592523081f024e1a976a2d02808f79b4. Latest consensus state: current_term: 1 leader_uuid: "592523081f024e1a976a2d02808f79b4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "508ed3a686554bd6af0204840904ab09" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44497 } } peers { permanent_uuid: "7a5f5a1ce0964d918460512bd4a094d1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42367 } } peers { permanent_uuid: "592523081f024e1a976a2d02808f79b4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33863 } } }
I20260924 21:08:07.678447 2627 tablet_replica.cc:442] T ec9a7eb6eb89431dae5e4e8bf3f597f3 P 592523081f024e1a976a2d02808f79b4: This TxnStatusTablet replica's current role is: LEADER
I20260924 21:08:07.678653 2639 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260924 21:08:07.678629 2113 catalog_manager.cc:5495] T ec9a7eb6eb89431dae5e4e8bf3f597f3 P 592523081f024e1a976a2d02808f79b4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 592523081f024e1a976a2d02808f79b4 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "592523081f024e1a976a2d02808f79b4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "508ed3a686554bd6af0204840904ab09" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44497 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a5f5a1ce0964d918460512bd4a094d1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42367 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "592523081f024e1a976a2d02808f79b4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33863 } health_report { overall_health: HEALTHY } } }
I20260924 21:08:07.678779 2639 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260924 21:08:07.697621 2634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:07.694783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33863 (local address 127.0.0.1:53756)
0924 21:08:07.694864 (+ 81us) negotiation.cc:107] Waiting for socket to connect
0924 21:08:07.694911 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0924 21:08:07.694958 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 21:08:07.695339 (+ 381us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 21:08:07.695339 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 21:08:07.695340 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 21:08:07.695379 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:07.695381 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:07.696164 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:07.696164 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:07.696665 (+ 501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:07.696666 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:07.697152 (+ 486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:07.697153 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:07.697202 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:07.697215 (+ 13us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 21:08:07.697480 (+ 265us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 21:08:07.697486 (+ 6us) client_negotiation.cc:769] Sending connection context
0924 21:08:07.697523 (+ 37us) client_negotiation.cc:240] Negotiation successful
0924 21:08:07.697570 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47}
I20260924 21:08:07.697619 2615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:07.695141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53756 (local address 127.0.98.3:33863)
0924 21:08:07.695196 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0924 21:08:07.695198 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 21:08:07.695206 (+ 8us) server_negotiation.cc:407] Connection header received
0924 21:08:07.695241 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:08:07.695242 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:08:07.695259 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:08:07.695322 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 21:08:07.695437 (+ 115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:07.696097 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:07.696732 (+ 635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:07.697070 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:07.697140 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:07.697301 (+ 161us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 21:08:07.697412 (+ 111us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 21:08:07.697490 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0924 21:08:07.697528 (+ 38us) server_negotiation.cc:299] Negotiation successful
0924 21:08:07.697549 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260924 21:08:07.699245 2271 raft_consensus.cc:1261] T ec9a7eb6eb89431dae5e4e8bf3f597f3 P 508ed3a686554bd6af0204840904ab09 [term 1 FOLLOWER]: Refusing update from remote peer 592523081f024e1a976a2d02808f79b4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 21:08:07.699262 2411 raft_consensus.cc:1261] T ec9a7eb6eb89431dae5e4e8bf3f597f3 P 7a5f5a1ce0964d918460512bd4a094d1 [term 1 FOLLOWER]: Refusing update from remote peer 592523081f024e1a976a2d02808f79b4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 21:08:07.699515 2627 consensus_queue.cc:1035] T ec9a7eb6eb89431dae5e4e8bf3f597f3 P 592523081f024e1a976a2d02808f79b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a5f5a1ce0964d918460512bd4a094d1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42367 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 21:08:07.699633 2624 consensus_queue.cc:1035] T ec9a7eb6eb89431dae5e4e8bf3f597f3 P 592523081f024e1a976a2d02808f79b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "508ed3a686554bd6af0204840904ab09" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44497 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 21:08:07.700594 2625 tablet_replica.cc:439] T ec9a7eb6eb89431dae5e4e8bf3f597f3 P 508ed3a686554bd6af0204840904ab09: TxnStatusTablet state changed. Reason: New leader 592523081f024e1a976a2d02808f79b4. Latest consensus state: current_term: 1 leader_uuid: "592523081f024e1a976a2d02808f79b4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "508ed3a686554bd6af0204840904ab09" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44497 } } peers { permanent_uuid: "7a5f5a1ce0964d918460512bd4a094d1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42367 } } peers { permanent_uuid: "592523081f024e1a976a2d02808f79b4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33863 } } }
I20260924 21:08:07.700668 2625 tablet_replica.cc:442] T ec9a7eb6eb89431dae5e4e8bf3f597f3 P 508ed3a686554bd6af0204840904ab09: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 21:08:07.700680 2623 tablet_replica.cc:439] T ec9a7eb6eb89431dae5e4e8bf3f597f3 P 7a5f5a1ce0964d918460512bd4a094d1: TxnStatusTablet state changed. Reason: New leader 592523081f024e1a976a2d02808f79b4. Latest consensus state: current_term: 1 leader_uuid: "592523081f024e1a976a2d02808f79b4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "508ed3a686554bd6af0204840904ab09" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44497 } } peers { permanent_uuid: "7a5f5a1ce0964d918460512bd4a094d1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42367 } } peers { permanent_uuid: "592523081f024e1a976a2d02808f79b4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33863 } } }
I20260924 21:08:07.700764 2623 tablet_replica.cc:442] T ec9a7eb6eb89431dae5e4e8bf3f597f3 P 7a5f5a1ce0964d918460512bd4a094d1: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 21:08:07.701722 2642 mvcc.cc:205] Tried to move back new op lower bound from 7333003623214071808 to 7333003623129223168. Current Snapshot: MvccSnapshot[applied={T|T < 7333003623214071808}]
I20260924 21:08:07.701886 2627 tablet_replica.cc:439] T ec9a7eb6eb89431dae5e4e8bf3f597f3 P 592523081f024e1a976a2d02808f79b4: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "592523081f024e1a976a2d02808f79b4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "508ed3a686554bd6af0204840904ab09" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44497 } } peers { permanent_uuid: "7a5f5a1ce0964d918460512bd4a094d1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42367 } } peers { permanent_uuid: "592523081f024e1a976a2d02808f79b4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33863 } } }
I20260924 21:08:07.701967 2624 tablet_replica.cc:439] T ec9a7eb6eb89431dae5e4e8bf3f597f3 P 592523081f024e1a976a2d02808f79b4: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "592523081f024e1a976a2d02808f79b4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "508ed3a686554bd6af0204840904ab09" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44497 } } peers { permanent_uuid: "7a5f5a1ce0964d918460512bd4a094d1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42367 } } peers { permanent_uuid: "592523081f024e1a976a2d02808f79b4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33863 } } }
I20260924 21:08:07.702018 2624 tablet_replica.cc:442] T ec9a7eb6eb89431dae5e4e8bf3f597f3 P 592523081f024e1a976a2d02808f79b4: This TxnStatusTablet replica's current role is: LEADER
I20260924 21:08:07.702064 2625 tablet_replica.cc:439] T ec9a7eb6eb89431dae5e4e8bf3f597f3 P 508ed3a686554bd6af0204840904ab09: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "592523081f024e1a976a2d02808f79b4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "508ed3a686554bd6af0204840904ab09" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44497 } } peers { permanent_uuid: "7a5f5a1ce0964d918460512bd4a094d1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42367 } } peers { permanent_uuid: "592523081f024e1a976a2d02808f79b4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33863 } } }
I20260924 21:08:07.702138 2625 tablet_replica.cc:442] T ec9a7eb6eb89431dae5e4e8bf3f597f3 P 508ed3a686554bd6af0204840904ab09: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 21:08:07.701957 2627 tablet_replica.cc:442] T ec9a7eb6eb89431dae5e4e8bf3f597f3 P 592523081f024e1a976a2d02808f79b4: This TxnStatusTablet replica's current role is: LEADER
I20260924 21:08:07.702401 2623 tablet_replica.cc:439] T ec9a7eb6eb89431dae5e4e8bf3f597f3 P 7a5f5a1ce0964d918460512bd4a094d1: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "592523081f024e1a976a2d02808f79b4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "508ed3a686554bd6af0204840904ab09" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44497 } } peers { permanent_uuid: "7a5f5a1ce0964d918460512bd4a094d1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42367 } } peers { permanent_uuid: "592523081f024e1a976a2d02808f79b4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33863 } } }
I20260924 21:08:07.702472 2623 tablet_replica.cc:442] T ec9a7eb6eb89431dae5e4e8bf3f597f3 P 7a5f5a1ce0964d918460512bd4a094d1: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 21:08:07.703164 2643 mvcc.cc:205] Tried to move back new op lower bound from 7333003623214071808 to 7333003623129223168. Current Snapshot: MvccSnapshot[applied={T|T < 7333003623214071808 or (T in {7333003623214071808})}]
I20260924 21:08:07.703258 2644 mvcc.cc:205] Tried to move back new op lower bound from 7333003623214071808 to 7333003623129223168. Current Snapshot: MvccSnapshot[applied={T|T < 7333003623214071808}]
I20260924 21:08:07.706125 2615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:07.703997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53762 (local address 127.0.98.3:33863)
0924 21:08:07.704040 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0924 21:08:07.704041 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 21:08:07.704048 (+ 7us) server_negotiation.cc:407] Connection header received
0924 21:08:07.704103 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:08:07.704104 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:08:07.704110 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:08:07.704164 (+ 54us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 21:08:07.704276 (+ 112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:07.704703 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:07.705467 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:07.705680 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:07.705771 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:07.705918 (+ 147us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 21:08:07.705944 (+ 26us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 21:08:07.705991 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0924 21:08:07.706045 (+ 54us) server_negotiation.cc:299] Negotiation successful
0924 21:08:07.706061 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260924 21:08:07.710589 2603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:07.708369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33863 (local address 127.0.98.3:35801)
0924 21:08:07.708414 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0924 21:08:07.708422 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0924 21:08:07.708459 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 21:08:07.708558 (+ 99us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 21:08:07.708558 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 21:08:07.708559 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 21:08:07.708616 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:07.708618 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:07.709314 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:07.709314 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:07.709798 (+ 484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:07.709800 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:07.710263 (+ 463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:07.710264 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:07.710306 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:07.710317 (+ 11us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 21:08:07.710479 (+ 162us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 21:08:07.710480 (+ 1us) client_negotiation.cc:769] Sending connection context
0924 21:08:07.710510 (+ 30us) client_negotiation.cc:240] Negotiation successful
0924 21:08:07.710537 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18}
I20260924 21:08:07.710639 2615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:07.708402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:35801 (local address 127.0.98.3:33863)
0924 21:08:07.708433 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0924 21:08:07.708434 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 21:08:07.708446 (+ 12us) server_negotiation.cc:407] Connection header received
0924 21:08:07.708494 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:08:07.708495 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:08:07.708502 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:08:07.708556 (+ 54us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 21:08:07.708672 (+ 116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:07.709254 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:07.709910 (+ 656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:07.710198 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:07.710260 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:07.710395 (+ 135us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 21:08:07.710423 (+ 28us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 21:08:07.710474 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0924 21:08:07.710521 (+ 47us) server_negotiation.cc:299] Negotiation successful
0924 21:08:07.710539 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260924 21:08:07.712399 2411 raft_consensus.cc:1261] T 2406f074aab3424782d86d6badbcca51 P 7a5f5a1ce0964d918460512bd4a094d1 [term 1 FOLLOWER]: Refusing update from remote peer 592523081f024e1a976a2d02808f79b4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 21:08:07.712410 2271 raft_consensus.cc:1261] T 2406f074aab3424782d86d6badbcca51 P 508ed3a686554bd6af0204840904ab09 [term 1 FOLLOWER]: Refusing update from remote peer 592523081f024e1a976a2d02808f79b4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 21:08:07.712561 2647 consensus_queue.cc:1035] T 2406f074aab3424782d86d6badbcca51 P 592523081f024e1a976a2d02808f79b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a5f5a1ce0964d918460512bd4a094d1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42367 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 21:08:07.712634 2624 consensus_queue.cc:1035] T 2406f074aab3424782d86d6badbcca51 P 592523081f024e1a976a2d02808f79b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "508ed3a686554bd6af0204840904ab09" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44497 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 21:08:07.732437 2113 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57948:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 21:08:07.732525 2113 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57948:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 21:08:07.733189 2113 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 224fefe284af480497048ba6af0e88cf: Sending DeleteTablet for 3 replicas of tablet 2406f074aab3424782d86d6badbcca51
I20260924 21:08:07.733501 2391 tablet_service.cc:1514] Processing DeleteTablet for tablet 2406f074aab3424782d86d6badbcca51 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 21:08:07 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:45502
I20260924 21:08:07.733548 2251 tablet_service.cc:1514] Processing DeleteTablet for tablet 2406f074aab3424782d86d6badbcca51 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 21:08:07 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:43754
I20260924 21:08:07.733587 2530 tablet_service.cc:1514] Processing DeleteTablet for tablet 2406f074aab3424782d86d6badbcca51 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 21:08:07 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:53752
I20260924 21:08:07.733814 2658 tablet_replica.cc:330] T 2406f074aab3424782d86d6badbcca51 P 508ed3a686554bd6af0204840904ab09: stopping tablet replica
I20260924 21:08:07.733853 2657 tablet_replica.cc:330] T 2406f074aab3424782d86d6badbcca51 P 7a5f5a1ce0964d918460512bd4a094d1: stopping tablet replica
I20260924 21:08:07.733870 2113 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57948:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260924 21:08:07.733932 2658 raft_consensus.cc:2229] T 2406f074aab3424782d86d6badbcca51 P 508ed3a686554bd6af0204840904ab09 [term 1 FOLLOWER]: Raft consensus shutting down.
W20260924 21:08:07.733997 2113 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 21:08:07.734002 2658 raft_consensus.cc:2258] T 2406f074aab3424782d86d6badbcca51 P 508ed3a686554bd6af0204840904ab09 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 21:08:07.734135 2659 tablet_replica.cc:330] T 2406f074aab3424782d86d6badbcca51 P 592523081f024e1a976a2d02808f79b4: stopping tablet replica
I20260924 21:08:07.734223 2659 raft_consensus.cc:2229] T 2406f074aab3424782d86d6badbcca51 P 592523081f024e1a976a2d02808f79b4 [term 1 LEADER]: Raft consensus shutting down.
I20260924 21:08:07.734298 2659 raft_consensus.cc:2258] T 2406f074aab3424782d86d6badbcca51 P 592523081f024e1a976a2d02808f79b4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 21:08:07.734627 2659 ts_tablet_manager.cc:1905] T 2406f074aab3424782d86d6badbcca51 P 592523081f024e1a976a2d02808f79b4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 21:08:07.735028 2391 tablet_service.cc:1467] Processing CreateTablet for tablet a5189aaacef14000892f5fcc8e7209d8 (DEFAULT_TABLE table=test-table [id=b0d6fc0f84024c56a64bbf60020a68f0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 21:08:07.735143 2391 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a5189aaacef14000892f5fcc8e7209d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 21:08:07.735148 2251 tablet_service.cc:1467] Processing CreateTablet for tablet a5189aaacef14000892f5fcc8e7209d8 (DEFAULT_TABLE table=test-table [id=b0d6fc0f84024c56a64bbf60020a68f0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 21:08:07.735164 2530 tablet_service.cc:1467] Processing CreateTablet for tablet a5189aaacef14000892f5fcc8e7209d8 (DEFAULT_TABLE table=test-table [id=b0d6fc0f84024c56a64bbf60020a68f0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 21:08:07.735245 2251 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a5189aaacef14000892f5fcc8e7209d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 21:08:07.735245 2530 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a5189aaacef14000892f5fcc8e7209d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 21:08:07.736265 2659 ts_tablet_manager.cc:1918] T 2406f074aab3424782d86d6badbcca51 P 592523081f024e1a976a2d02808f79b4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 21:08:07.736323 2659 log.cc:1196] T 2406f074aab3424782d86d6badbcca51 P 592523081f024e1a976a2d02808f79b4: Deleting WAL directory at /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790284076800039-392-0/minicluster-data/ts-2/wal/wals/2406f074aab3424782d86d6badbcca51
I20260924 21:08:07.736505 2617 tablet_bootstrap.cc:492] T a5189aaacef14000892f5fcc8e7209d8 P 7a5f5a1ce0964d918460512bd4a094d1: Bootstrap starting.
I20260924 21:08:07.736538 2618 tablet_bootstrap.cc:492] T a5189aaacef14000892f5fcc8e7209d8 P 592523081f024e1a976a2d02808f79b4: Bootstrap starting.
I20260924 21:08:07.736537 2621 tablet_bootstrap.cc:492] T a5189aaacef14000892f5fcc8e7209d8 P 508ed3a686554bd6af0204840904ab09: Bootstrap starting.
I20260924 21:08:07.736910 2617 tablet_bootstrap.cc:654] T a5189aaacef14000892f5fcc8e7209d8 P 7a5f5a1ce0964d918460512bd4a094d1: Neither blocks nor log segments found. Creating new log.
I20260924 21:08:07.736914 2618 tablet_bootstrap.cc:654] T a5189aaacef14000892f5fcc8e7209d8 P 592523081f024e1a976a2d02808f79b4: Neither blocks nor log segments found. Creating new log.
I20260924 21:08:07.736936 2621 tablet_bootstrap.cc:654] T a5189aaacef14000892f5fcc8e7209d8 P 508ed3a686554bd6af0204840904ab09: Neither blocks nor log segments found. Creating new log.
I20260924 21:08:07.736948 2658 ts_tablet_manager.cc:1905] T 2406f074aab3424782d86d6badbcca51 P 508ed3a686554bd6af0204840904ab09: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 21:08:07.737751 2659 ts_tablet_manager.cc:1939] T 2406f074aab3424782d86d6badbcca51 P 592523081f024e1a976a2d02808f79b4: Deleting consensus metadata
I20260924 21:08:07.734401 2657 raft_consensus.cc:2229] T 2406f074aab3424782d86d6badbcca51 P 7a5f5a1ce0964d918460512bd4a094d1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 21:08:07.738162 2098 catalog_manager.cc:4841] TS 592523081f024e1a976a2d02808f79b4 (127.0.98.3:33863): tablet 2406f074aab3424782d86d6badbcca51 (table test-table [id=1fb1a04f719b48b99bf24be0738784aa]) successfully deleted
I20260924 21:08:07.738279 2657 raft_consensus.cc:2258] T 2406f074aab3424782d86d6badbcca51 P 7a5f5a1ce0964d918460512bd4a094d1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 21:08:07.738340 2658 ts_tablet_manager.cc:1918] T 2406f074aab3424782d86d6badbcca51 P 508ed3a686554bd6af0204840904ab09: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 21:08:07.738408 2658 log.cc:1196] T 2406f074aab3424782d86d6badbcca51 P 508ed3a686554bd6af0204840904ab09: Deleting WAL directory at /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790284076800039-392-0/minicluster-data/ts-0/wal/wals/2406f074aab3424782d86d6badbcca51
I20260924 21:08:07.738541 2621 tablet_bootstrap.cc:492] T a5189aaacef14000892f5fcc8e7209d8 P 508ed3a686554bd6af0204840904ab09: No bootstrap required, opened a new log
I20260924 21:08:07.738587 2621 ts_tablet_manager.cc:1397] T a5189aaacef14000892f5fcc8e7209d8 P 508ed3a686554bd6af0204840904ab09: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 21:08:07.738641 2658 ts_tablet_manager.cc:1939] T 2406f074aab3424782d86d6badbcca51 P 508ed3a686554bd6af0204840904ab09: Deleting consensus metadata
I20260924 21:08:07.738754 2617 tablet_bootstrap.cc:492] T a5189aaacef14000892f5fcc8e7209d8 P 7a5f5a1ce0964d918460512bd4a094d1: No bootstrap required, opened a new log
I20260924 21:08:07.738729 2621 raft_consensus.cc:348] T a5189aaacef14000892f5fcc8e7209d8 P 508ed3a686554bd6af0204840904ab09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "508ed3a686554bd6af0204840904ab09" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44497 } } peers { permanent_uuid: "7a5f5a1ce0964d918460512bd4a094d1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42367 } } peers { permanent_uuid: "592523081f024e1a976a2d02808f79b4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33863 } }
I20260924 21:08:07.738786 2621 raft_consensus.cc:374] T a5189aaacef14000892f5fcc8e7209d8 P 508ed3a686554bd6af0204840904ab09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 21:08:07.738796 2617 ts_tablet_manager.cc:1397] T a5189aaacef14000892f5fcc8e7209d8 P 7a5f5a1ce0964d918460512bd4a094d1: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260924 21:08:07.738811 2621 raft_consensus.cc:729] T a5189aaacef14000892f5fcc8e7209d8 P 508ed3a686554bd6af0204840904ab09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 508ed3a686554bd6af0204840904ab09, State: Initialized, Role: FOLLOWER
I20260924 21:08:07.738847 2621 consensus_queue.cc:260] T a5189aaacef14000892f5fcc8e7209d8 P 508ed3a686554bd6af0204840904ab09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "508ed3a686554bd6af0204840904ab09" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44497 } } peers { permanent_uuid: "7a5f5a1ce0964d918460512bd4a094d1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42367 } } peers { permanent_uuid: "592523081f024e1a976a2d02808f79b4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33863 } }
I20260924 21:08:07.738925 2621 ts_tablet_manager.cc:1428] T a5189aaacef14000892f5fcc8e7209d8 P 508ed3a686554bd6af0204840904ab09: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 21:08:07.738912 2617 raft_consensus.cc:348] T a5189aaacef14000892f5fcc8e7209d8 P 7a5f5a1ce0964d918460512bd4a094d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "508ed3a686554bd6af0204840904ab09" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44497 } } peers { permanent_uuid: "7a5f5a1ce0964d918460512bd4a094d1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42367 } } peers { permanent_uuid: "592523081f024e1a976a2d02808f79b4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33863 } }
I20260924 21:08:07.738965 2617 raft_consensus.cc:374] T a5189aaacef14000892f5fcc8e7209d8 P 7a5f5a1ce0964d918460512bd4a094d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 21:08:07.738989 2617 raft_consensus.cc:729] T a5189aaacef14000892f5fcc8e7209d8 P 7a5f5a1ce0964d918460512bd4a094d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a5f5a1ce0964d918460512bd4a094d1, State: Initialized, Role: FOLLOWER
I20260924 21:08:07.739032 2617 consensus_queue.cc:260] T a5189aaacef14000892f5fcc8e7209d8 P 7a5f5a1ce0964d918460512bd4a094d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "508ed3a686554bd6af0204840904ab09" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44497 } } peers { permanent_uuid: "7a5f5a1ce0964d918460512bd4a094d1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42367 } } peers { permanent_uuid: "592523081f024e1a976a2d02808f79b4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33863 } }
I20260924 21:08:07.739113 2617 ts_tablet_manager.cc:1428] T a5189aaacef14000892f5fcc8e7209d8 P 7a5f5a1ce0964d918460512bd4a094d1: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 21:08:07.739193 2618 tablet_bootstrap.cc:492] T a5189aaacef14000892f5fcc8e7209d8 P 592523081f024e1a976a2d02808f79b4: No bootstrap required, opened a new log
I20260924 21:08:07.739228 2618 ts_tablet_manager.cc:1397] T a5189aaacef14000892f5fcc8e7209d8 P 592523081f024e1a976a2d02808f79b4: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20260924 21:08:07.739231 2101 catalog_manager.cc:4841] TS 508ed3a686554bd6af0204840904ab09 (127.0.98.1:44497): tablet 2406f074aab3424782d86d6badbcca51 (table test-table [id=1fb1a04f719b48b99bf24be0738784aa]) successfully deleted
I20260924 21:08:07.739341 2618 raft_consensus.cc:348] T a5189aaacef14000892f5fcc8e7209d8 P 592523081f024e1a976a2d02808f79b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "508ed3a686554bd6af0204840904ab09" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44497 } } peers { permanent_uuid: "7a5f5a1ce0964d918460512bd4a094d1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42367 } } peers { permanent_uuid: "592523081f024e1a976a2d02808f79b4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33863 } }
I20260924 21:08:07.739395 2618 raft_consensus.cc:374] T a5189aaacef14000892f5fcc8e7209d8 P 592523081f024e1a976a2d02808f79b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 21:08:07.739418 2618 raft_consensus.cc:729] T a5189aaacef14000892f5fcc8e7209d8 P 592523081f024e1a976a2d02808f79b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 592523081f024e1a976a2d02808f79b4, State: Initialized, Role: FOLLOWER
I20260924 21:08:07.739455 2618 consensus_queue.cc:260] T a5189aaacef14000892f5fcc8e7209d8 P 592523081f024e1a976a2d02808f79b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "508ed3a686554bd6af0204840904ab09" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44497 } } peers { permanent_uuid: "7a5f5a1ce0964d918460512bd4a094d1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42367 } } peers { permanent_uuid: "592523081f024e1a976a2d02808f79b4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33863 } }
I20260924 21:08:07.739530 2618 ts_tablet_manager.cc:1428] T a5189aaacef14000892f5fcc8e7209d8 P 592523081f024e1a976a2d02808f79b4: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 21:08:07.739755 2657 ts_tablet_manager.cc:1905] T 2406f074aab3424782d86d6badbcca51 P 7a5f5a1ce0964d918460512bd4a094d1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 21:08:07.741255 2657 ts_tablet_manager.cc:1918] T 2406f074aab3424782d86d6badbcca51 P 7a5f5a1ce0964d918460512bd4a094d1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 21:08:07.741318 2657 log.cc:1196] T 2406f074aab3424782d86d6badbcca51 P 7a5f5a1ce0964d918460512bd4a094d1: Deleting WAL directory at /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790284076800039-392-0/minicluster-data/ts-1/wal/wals/2406f074aab3424782d86d6badbcca51
I20260924 21:08:07.741526 2657 ts_tablet_manager.cc:1939] T 2406f074aab3424782d86d6badbcca51 P 7a5f5a1ce0964d918460512bd4a094d1: Deleting consensus metadata
I20260924 21:08:07.741991 2099 catalog_manager.cc:4841] TS 7a5f5a1ce0964d918460512bd4a094d1 (127.0.98.2:42367): tablet 2406f074aab3424782d86d6badbcca51 (table test-table [id=1fb1a04f719b48b99bf24be0738784aa]) successfully deleted
I20260924 21:08:08.051697 2647 raft_consensus.cc:482] T a5189aaacef14000892f5fcc8e7209d8 P 592523081f024e1a976a2d02808f79b4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 21:08:08.051824 2647 raft_consensus.cc:504] T a5189aaacef14000892f5fcc8e7209d8 P 592523081f024e1a976a2d02808f79b4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "508ed3a686554bd6af0204840904ab09" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44497 } } peers { permanent_uuid: "7a5f5a1ce0964d918460512bd4a094d1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42367 } } peers { permanent_uuid: "592523081f024e1a976a2d02808f79b4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33863 } }
I20260924 21:08:08.051976 2647 leader_election.cc:288] T a5189aaacef14000892f5fcc8e7209d8 P 592523081f024e1a976a2d02808f79b4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 508ed3a686554bd6af0204840904ab09 (127.0.98.1:44497), 7a5f5a1ce0964d918460512bd4a094d1 (127.0.98.2:42367)
I20260924 21:08:08.052175 2411 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5189aaacef14000892f5fcc8e7209d8" candidate_uuid: "592523081f024e1a976a2d02808f79b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a5f5a1ce0964d918460512bd4a094d1" is_pre_election: true
I20260924 21:08:08.052175 2271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5189aaacef14000892f5fcc8e7209d8" candidate_uuid: "592523081f024e1a976a2d02808f79b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "508ed3a686554bd6af0204840904ab09" is_pre_election: true
I20260924 21:08:08.052276 2271 raft_consensus.cc:2454] T a5189aaacef14000892f5fcc8e7209d8 P 508ed3a686554bd6af0204840904ab09 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 592523081f024e1a976a2d02808f79b4 in term 0.
I20260924 21:08:08.052275 2411 raft_consensus.cc:2454] T a5189aaacef14000892f5fcc8e7209d8 P 7a5f5a1ce0964d918460512bd4a094d1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 592523081f024e1a976a2d02808f79b4 in term 0.
I20260924 21:08:08.052452 2483 leader_election.cc:302] T a5189aaacef14000892f5fcc8e7209d8 P 592523081f024e1a976a2d02808f79b4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 592523081f024e1a976a2d02808f79b4, 7a5f5a1ce0964d918460512bd4a094d1; no voters:
I20260924 21:08:08.052561 2647 raft_consensus.cc:2781] T a5189aaacef14000892f5fcc8e7209d8 P 592523081f024e1a976a2d02808f79b4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 21:08:08.052601 2647 raft_consensus.cc:482] T a5189aaacef14000892f5fcc8e7209d8 P 592523081f024e1a976a2d02808f79b4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 21:08:08.052626 2647 raft_consensus.cc:3037] T a5189aaacef14000892f5fcc8e7209d8 P 592523081f024e1a976a2d02808f79b4 [term 0 FOLLOWER]: Advancing to term 1
I20260924 21:08:08.053326 2647 raft_consensus.cc:504] T a5189aaacef14000892f5fcc8e7209d8 P 592523081f024e1a976a2d02808f79b4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "508ed3a686554bd6af0204840904ab09" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44497 } } peers { permanent_uuid: "7a5f5a1ce0964d918460512bd4a094d1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42367 } } peers { permanent_uuid: "592523081f024e1a976a2d02808f79b4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33863 } }
I20260924 21:08:08.053465 2647 leader_election.cc:288] T a5189aaacef14000892f5fcc8e7209d8 P 592523081f024e1a976a2d02808f79b4 [CANDIDATE]: Term 1 election: Requested vote from peers 508ed3a686554bd6af0204840904ab09 (127.0.98.1:44497), 7a5f5a1ce0964d918460512bd4a094d1 (127.0.98.2:42367)
I20260924 21:08:08.053658 2271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5189aaacef14000892f5fcc8e7209d8" candidate_uuid: "592523081f024e1a976a2d02808f79b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "508ed3a686554bd6af0204840904ab09"
I20260924 21:08:08.053665 2411 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5189aaacef14000892f5fcc8e7209d8" candidate_uuid: "592523081f024e1a976a2d02808f79b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a5f5a1ce0964d918460512bd4a094d1"
I20260924 21:08:08.053745 2411 raft_consensus.cc:3037] T a5189aaacef14000892f5fcc8e7209d8 P 7a5f5a1ce0964d918460512bd4a094d1 [term 0 FOLLOWER]: Advancing to term 1
I20260924 21:08:08.053735 2271 raft_consensus.cc:3037] T a5189aaacef14000892f5fcc8e7209d8 P 508ed3a686554bd6af0204840904ab09 [term 0 FOLLOWER]: Advancing to term 1
I20260924 21:08:08.054531 2411 raft_consensus.cc:2454] T a5189aaacef14000892f5fcc8e7209d8 P 7a5f5a1ce0964d918460512bd4a094d1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 592523081f024e1a976a2d02808f79b4 in term 1.
I20260924 21:08:08.054531 2271 raft_consensus.cc:2454] T a5189aaacef14000892f5fcc8e7209d8 P 508ed3a686554bd6af0204840904ab09 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 592523081f024e1a976a2d02808f79b4 in term 1.
I20260924 21:08:08.054719 2485 leader_election.cc:302] T a5189aaacef14000892f5fcc8e7209d8 P 592523081f024e1a976a2d02808f79b4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 508ed3a686554bd6af0204840904ab09, 592523081f024e1a976a2d02808f79b4; no voters:
I20260924 21:08:08.054839 2647 raft_consensus.cc:2781] T a5189aaacef14000892f5fcc8e7209d8 P 592523081f024e1a976a2d02808f79b4 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 21:08:08.054924 2647 raft_consensus.cc:686] T a5189aaacef14000892f5fcc8e7209d8 P 592523081f024e1a976a2d02808f79b4 [term 1 LEADER]: Becoming Leader. State: Replica: 592523081f024e1a976a2d02808f79b4, State: Running, Role: LEADER
I20260924 21:08:08.054971 2647 consensus_queue.cc:237] T a5189aaacef14000892f5fcc8e7209d8 P 592523081f024e1a976a2d02808f79b4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "508ed3a686554bd6af0204840904ab09" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44497 } } peers { permanent_uuid: "7a5f5a1ce0964d918460512bd4a094d1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42367 } } peers { permanent_uuid: "592523081f024e1a976a2d02808f79b4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33863 } }
I20260924 21:08:08.055608 2114 catalog_manager.cc:5495] T a5189aaacef14000892f5fcc8e7209d8 P 592523081f024e1a976a2d02808f79b4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 592523081f024e1a976a2d02808f79b4 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "592523081f024e1a976a2d02808f79b4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "508ed3a686554bd6af0204840904ab09" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44497 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a5f5a1ce0964d918460512bd4a094d1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42367 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "592523081f024e1a976a2d02808f79b4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33863 } health_report { overall_health: HEALTHY } } }
I20260924 21:08:08.092149 2271 raft_consensus.cc:1261] T a5189aaacef14000892f5fcc8e7209d8 P 508ed3a686554bd6af0204840904ab09 [term 1 FOLLOWER]: Refusing update from remote peer 592523081f024e1a976a2d02808f79b4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 21:08:08.092139 2411 raft_consensus.cc:1261] T a5189aaacef14000892f5fcc8e7209d8 P 7a5f5a1ce0964d918460512bd4a094d1 [term 1 FOLLOWER]: Refusing update from remote peer 592523081f024e1a976a2d02808f79b4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 21:08:08.092357 2627 consensus_queue.cc:1035] T a5189aaacef14000892f5fcc8e7209d8 P 592523081f024e1a976a2d02808f79b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "508ed3a686554bd6af0204840904ab09" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44497 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 21:08:08.092445 2647 consensus_queue.cc:1035] T a5189aaacef14000892f5fcc8e7209d8 P 592523081f024e1a976a2d02808f79b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a5f5a1ce0964d918460512bd4a094d1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42367 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260924 21:08:08.096388 2642 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260924 21:08:09.611797 2113 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57948:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 21:08:09.611884 2113 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57948:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 21:08:09.612660 2113 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 224fefe284af480497048ba6af0e88cf: Sending DeleteTablet for 3 replicas of tablet a5189aaacef14000892f5fcc8e7209d8
I20260924 21:08:09.612900 2391 tablet_service.cc:1514] Processing DeleteTablet for tablet a5189aaacef14000892f5fcc8e7209d8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 21:08:09 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:45502
I20260924 21:08:09.612906 2531 tablet_service.cc:1514] Processing DeleteTablet for tablet a5189aaacef14000892f5fcc8e7209d8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 21:08:09 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:53752
I20260924 21:08:09.613062 2251 tablet_service.cc:1514] Processing DeleteTablet for tablet a5189aaacef14000892f5fcc8e7209d8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 21:08:09 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:43754
I20260924 21:08:09.613229 2687 tablet_replica.cc:330] T a5189aaacef14000892f5fcc8e7209d8 P 7a5f5a1ce0964d918460512bd4a094d1: stopping tablet replica
I20260924 21:08:09.613229 2689 tablet_replica.cc:330] T a5189aaacef14000892f5fcc8e7209d8 P 508ed3a686554bd6af0204840904ab09: stopping tablet replica
I20260924 21:08:09.613287 2687 raft_consensus.cc:2229] T a5189aaacef14000892f5fcc8e7209d8 P 7a5f5a1ce0964d918460512bd4a094d1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 21:08:09.613300 2689 raft_consensus.cc:2229] T a5189aaacef14000892f5fcc8e7209d8 P 508ed3a686554bd6af0204840904ab09 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 21:08:09.613327 2687 raft_consensus.cc:2258] T a5189aaacef14000892f5fcc8e7209d8 P 7a5f5a1ce0964d918460512bd4a094d1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 21:08:09.613334 2689 raft_consensus.cc:2258] T a5189aaacef14000892f5fcc8e7209d8 P 508ed3a686554bd6af0204840904ab09 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 21:08:09.613528 2689 ts_tablet_manager.cc:1905] T a5189aaacef14000892f5fcc8e7209d8 P 508ed3a686554bd6af0204840904ab09: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 21:08:09.614981 2689 ts_tablet_manager.cc:1918] T a5189aaacef14000892f5fcc8e7209d8 P 508ed3a686554bd6af0204840904ab09: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260924 21:08:09.615044 2689 log.cc:1196] T a5189aaacef14000892f5fcc8e7209d8 P 508ed3a686554bd6af0204840904ab09: Deleting WAL directory at /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790284076800039-392-0/minicluster-data/ts-0/wal/wals/a5189aaacef14000892f5fcc8e7209d8
I20260924 21:08:09.615250 2689 ts_tablet_manager.cc:1939] T a5189aaacef14000892f5fcc8e7209d8 P 508ed3a686554bd6af0204840904ab09: Deleting consensus metadata
I20260924 21:08:09.615542 2101 catalog_manager.cc:4841] TS 508ed3a686554bd6af0204840904ab09 (127.0.98.1:44497): tablet a5189aaacef14000892f5fcc8e7209d8 (table test-table [id=b0d6fc0f84024c56a64bbf60020a68f0]) successfully deleted
I20260924 21:08:09.615708 2687 ts_tablet_manager.cc:1905] T a5189aaacef14000892f5fcc8e7209d8 P 7a5f5a1ce0964d918460512bd4a094d1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 21:08:09.617355 2687 ts_tablet_manager.cc:1918] T a5189aaacef14000892f5fcc8e7209d8 P 7a5f5a1ce0964d918460512bd4a094d1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260924 21:08:09.617419 2687 log.cc:1196] T a5189aaacef14000892f5fcc8e7209d8 P 7a5f5a1ce0964d918460512bd4a094d1: Deleting WAL directory at /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790284076800039-392-0/minicluster-data/ts-1/wal/wals/a5189aaacef14000892f5fcc8e7209d8
I20260924 21:08:09.617653 2687 ts_tablet_manager.cc:1939] T a5189aaacef14000892f5fcc8e7209d8 P 7a5f5a1ce0964d918460512bd4a094d1: Deleting consensus metadata
I20260924 21:08:09.617924 2688 tablet_replica.cc:330] T a5189aaacef14000892f5fcc8e7209d8 P 592523081f024e1a976a2d02808f79b4: stopping tablet replica
I20260924 21:08:09.617965 2099 catalog_manager.cc:4841] TS 7a5f5a1ce0964d918460512bd4a094d1 (127.0.98.2:42367): tablet a5189aaacef14000892f5fcc8e7209d8 (table test-table [id=b0d6fc0f84024c56a64bbf60020a68f0]) successfully deleted
I20260924 21:08:09.617992 2688 raft_consensus.cc:2229] T a5189aaacef14000892f5fcc8e7209d8 P 592523081f024e1a976a2d02808f79b4 [term 1 LEADER]: Raft consensus shutting down.
I20260924 21:08:09.618052 2688 raft_consensus.cc:2258] T a5189aaacef14000892f5fcc8e7209d8 P 592523081f024e1a976a2d02808f79b4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 21:08:09.618288 2688 ts_tablet_manager.cc:1905] T a5189aaacef14000892f5fcc8e7209d8 P 592523081f024e1a976a2d02808f79b4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 21:08:09.619575 2688 ts_tablet_manager.cc:1918] T a5189aaacef14000892f5fcc8e7209d8 P 592523081f024e1a976a2d02808f79b4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260924 21:08:09.619632 2688 log.cc:1196] T a5189aaacef14000892f5fcc8e7209d8 P 592523081f024e1a976a2d02808f79b4: Deleting WAL directory at /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790284076800039-392-0/minicluster-data/ts-2/wal/wals/a5189aaacef14000892f5fcc8e7209d8
I20260924 21:08:09.619839 2688 ts_tablet_manager.cc:1939] T a5189aaacef14000892f5fcc8e7209d8 P 592523081f024e1a976a2d02808f79b4: Deleting consensus metadata
I20260924 21:08:09.620101 2098 catalog_manager.cc:4841] TS 592523081f024e1a976a2d02808f79b4 (127.0.98.3:33863): tablet a5189aaacef14000892f5fcc8e7209d8 (table test-table [id=b0d6fc0f84024c56a64bbf60020a68f0]) successfully deleted
Sep 24 21:08:09 dist-test-slave-c7tk krb5kdc[2069](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284089, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 24 21:08:09 dist-test-slave-c7tk krb5kdc[2069](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284089, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 21:08:09.641314 2697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:09.627878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57966 (local address 127.0.98.62:33355)
0924 21:08:09.628030 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0924 21:08:09.628032 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 21:08:09.628039 (+ 7us) server_negotiation.cc:407] Connection header received
0924 21:08:09.628130 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:08:09.628131 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:08:09.628144 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:08:09.628223 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:08:09.628708 (+ 485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:09.629161 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:09.629875 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:09.630070 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:09.630152 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:09.638861 (+ 8709us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 21:08:09.638863 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 21:08:09.638863 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 21:08:09.638869 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 21:08:09.640115 (+ 1246us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:08:09.640491 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:08:09.640492 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:08:09.640493 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:08:09.640554 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:08:09.640759 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:08:09.640760 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:08:09.640760 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:08:09.640891 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 21:08:09.641010 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0924 21:08:09.641131 (+ 121us) server_negotiation.cc:299] Negotiation successful
0924 21:08:09.641212 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":82,"threads_started":1}
I20260924 21:08:09.642556 2113 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57966:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 21:08:09.642694 2113 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 21:08:09.643983 2251 tablet_service.cc:1467] Processing CreateTablet for tablet 10b3b8d3228d4b78a0e4ca23a0cf3fee (DEFAULT_TABLE table=test-table [id=9c9328f2dfc0491db35053e14c573b31]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 21:08:09.644050 2531 tablet_service.cc:1467] Processing CreateTablet for tablet 10b3b8d3228d4b78a0e4ca23a0cf3fee (DEFAULT_TABLE table=test-table [id=9c9328f2dfc0491db35053e14c573b31]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 21:08:09.644138 2251 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 10b3b8d3228d4b78a0e4ca23a0cf3fee. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 21:08:09.644164 2531 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 10b3b8d3228d4b78a0e4ca23a0cf3fee. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 21:08:09.645190 2698 tablet_bootstrap.cc:492] T 10b3b8d3228d4b78a0e4ca23a0cf3fee P 508ed3a686554bd6af0204840904ab09: Bootstrap starting.
I20260924 21:08:09.645251 2391 tablet_service.cc:1467] Processing CreateTablet for tablet 10b3b8d3228d4b78a0e4ca23a0cf3fee (DEFAULT_TABLE table=test-table [id=9c9328f2dfc0491db35053e14c573b31]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 21:08:09.645220 2699 tablet_bootstrap.cc:492] T 10b3b8d3228d4b78a0e4ca23a0cf3fee P 592523081f024e1a976a2d02808f79b4: Bootstrap starting.
I20260924 21:08:09.645372 2391 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 10b3b8d3228d4b78a0e4ca23a0cf3fee. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 21:08:09.645506 2698 tablet_bootstrap.cc:654] T 10b3b8d3228d4b78a0e4ca23a0cf3fee P 508ed3a686554bd6af0204840904ab09: Neither blocks nor log segments found. Creating new log.
I20260924 21:08:09.645689 2699 tablet_bootstrap.cc:654] T 10b3b8d3228d4b78a0e4ca23a0cf3fee P 592523081f024e1a976a2d02808f79b4: Neither blocks nor log segments found. Creating new log.
I20260924 21:08:09.646204 2698 tablet_bootstrap.cc:492] T 10b3b8d3228d4b78a0e4ca23a0cf3fee P 508ed3a686554bd6af0204840904ab09: No bootstrap required, opened a new log
I20260924 21:08:09.646258 2698 ts_tablet_manager.cc:1397] T 10b3b8d3228d4b78a0e4ca23a0cf3fee P 508ed3a686554bd6af0204840904ab09: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260924 21:08:09.646449 2698 raft_consensus.cc:348] T 10b3b8d3228d4b78a0e4ca23a0cf3fee P 508ed3a686554bd6af0204840904ab09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "508ed3a686554bd6af0204840904ab09" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44497 } } peers { permanent_uuid: "7a5f5a1ce0964d918460512bd4a094d1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42367 } } peers { permanent_uuid: "592523081f024e1a976a2d02808f79b4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33863 } }
I20260924 21:08:09.646533 2698 raft_consensus.cc:374] T 10b3b8d3228d4b78a0e4ca23a0cf3fee P 508ed3a686554bd6af0204840904ab09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 21:08:09.646559 2698 raft_consensus.cc:729] T 10b3b8d3228d4b78a0e4ca23a0cf3fee P 508ed3a686554bd6af0204840904ab09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 508ed3a686554bd6af0204840904ab09, State: Initialized, Role: FOLLOWER
I20260924 21:08:09.646684 2698 consensus_queue.cc:260] T 10b3b8d3228d4b78a0e4ca23a0cf3fee P 508ed3a686554bd6af0204840904ab09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "508ed3a686554bd6af0204840904ab09" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44497 } } peers { permanent_uuid: "7a5f5a1ce0964d918460512bd4a094d1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42367 } } peers { permanent_uuid: "592523081f024e1a976a2d02808f79b4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33863 } }
I20260924 21:08:09.646725 2702 tablet_bootstrap.cc:492] T 10b3b8d3228d4b78a0e4ca23a0cf3fee P 7a5f5a1ce0964d918460512bd4a094d1: Bootstrap starting.
I20260924 21:08:09.646773 2703 raft_consensus.cc:482] T 10b3b8d3228d4b78a0e4ca23a0cf3fee P 508ed3a686554bd6af0204840904ab09 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 21:08:09.646819 2703 raft_consensus.cc:504] T 10b3b8d3228d4b78a0e4ca23a0cf3fee P 508ed3a686554bd6af0204840904ab09 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "508ed3a686554bd6af0204840904ab09" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44497 } } peers { permanent_uuid: "7a5f5a1ce0964d918460512bd4a094d1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42367 } } peers { permanent_uuid: "592523081f024e1a976a2d02808f79b4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33863 } }
I20260924 21:08:09.647048 2698 ts_tablet_manager.cc:1428] T 10b3b8d3228d4b78a0e4ca23a0cf3fee P 508ed3a686554bd6af0204840904ab09: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260924 21:08:09.647087 2702 tablet_bootstrap.cc:654] T 10b3b8d3228d4b78a0e4ca23a0cf3fee P 7a5f5a1ce0964d918460512bd4a094d1: Neither blocks nor log segments found. Creating new log.
I20260924 21:08:09.647280 2699 tablet_bootstrap.cc:492] T 10b3b8d3228d4b78a0e4ca23a0cf3fee P 592523081f024e1a976a2d02808f79b4: No bootstrap required, opened a new log
I20260924 21:08:09.647326 2699 ts_tablet_manager.cc:1397] T 10b3b8d3228d4b78a0e4ca23a0cf3fee P 592523081f024e1a976a2d02808f79b4: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 21:08:09.647437 2699 raft_consensus.cc:348] T 10b3b8d3228d4b78a0e4ca23a0cf3fee P 592523081f024e1a976a2d02808f79b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "508ed3a686554bd6af0204840904ab09" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44497 } } peers { permanent_uuid: "7a5f5a1ce0964d918460512bd4a094d1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42367 } } peers { permanent_uuid: "592523081f024e1a976a2d02808f79b4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33863 } }
I20260924 21:08:09.647518 2699 raft_consensus.cc:374] T 10b3b8d3228d4b78a0e4ca23a0cf3fee P 592523081f024e1a976a2d02808f79b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 21:08:09.647550 2699 raft_consensus.cc:729] T 10b3b8d3228d4b78a0e4ca23a0cf3fee P 592523081f024e1a976a2d02808f79b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 592523081f024e1a976a2d02808f79b4, State: Initialized, Role: FOLLOWER
I20260924 21:08:09.647607 2703 leader_election.cc:288] T 10b3b8d3228d4b78a0e4ca23a0cf3fee P 508ed3a686554bd6af0204840904ab09 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7a5f5a1ce0964d918460512bd4a094d1 (127.0.98.2:42367), 592523081f024e1a976a2d02808f79b4 (127.0.98.3:33863)
I20260924 21:08:09.647727 2699 consensus_queue.cc:260] T 10b3b8d3228d4b78a0e4ca23a0cf3fee P 592523081f024e1a976a2d02808f79b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "508ed3a686554bd6af0204840904ab09" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44497 } } peers { permanent_uuid: "7a5f5a1ce0964d918460512bd4a094d1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42367 } } peers { permanent_uuid: "592523081f024e1a976a2d02808f79b4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33863 } }
I20260924 21:08:09.647863 2699 ts_tablet_manager.cc:1428] T 10b3b8d3228d4b78a0e4ca23a0cf3fee P 592523081f024e1a976a2d02808f79b4: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260924 21:08:09.648119 2702 tablet_bootstrap.cc:492] T 10b3b8d3228d4b78a0e4ca23a0cf3fee P 7a5f5a1ce0964d918460512bd4a094d1: No bootstrap required, opened a new log
I20260924 21:08:09.648169 2702 ts_tablet_manager.cc:1397] T 10b3b8d3228d4b78a0e4ca23a0cf3fee P 7a5f5a1ce0964d918460512bd4a094d1: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260924 21:08:09.648309 2702 raft_consensus.cc:348] T 10b3b8d3228d4b78a0e4ca23a0cf3fee P 7a5f5a1ce0964d918460512bd4a094d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "508ed3a686554bd6af0204840904ab09" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44497 } } peers { permanent_uuid: "7a5f5a1ce0964d918460512bd4a094d1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42367 } } peers { permanent_uuid: "592523081f024e1a976a2d02808f79b4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33863 } }
I20260924 21:08:09.648376 2702 raft_consensus.cc:374] T 10b3b8d3228d4b78a0e4ca23a0cf3fee P 7a5f5a1ce0964d918460512bd4a094d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 21:08:09.648401 2702 raft_consensus.cc:729] T 10b3b8d3228d4b78a0e4ca23a0cf3fee P 7a5f5a1ce0964d918460512bd4a094d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a5f5a1ce0964d918460512bd4a094d1, State: Initialized, Role: FOLLOWER
I20260924 21:08:09.648458 2702 consensus_queue.cc:260] T 10b3b8d3228d4b78a0e4ca23a0cf3fee P 7a5f5a1ce0964d918460512bd4a094d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "508ed3a686554bd6af0204840904ab09" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44497 } } peers { permanent_uuid: "7a5f5a1ce0964d918460512bd4a094d1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42367 } } peers { permanent_uuid: "592523081f024e1a976a2d02808f79b4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33863 } }
I20260924 21:08:09.648598 2702 ts_tablet_manager.cc:1428] T 10b3b8d3228d4b78a0e4ca23a0cf3fee P 7a5f5a1ce0964d918460512bd4a094d1: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260924 21:08:09.650790 2706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:09.647316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:51553 (local address 127.0.98.2:42367)
0924 21:08:09.648014 (+ 698us) server_negotiation.cc:206] Beginning negotiation
0924 21:08:09.648016 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 21:08:09.648181 (+ 165us) server_negotiation.cc:407] Connection header received
0924 21:08:09.648217 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:08:09.648218 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:08:09.648229 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:08:09.648284 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 21:08:09.648471 (+ 187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:09.649070 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:09.650034 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:09.650515 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:09.650561 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:09.650568 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0924 21:08:09.650732 (+ 164us) server_negotiation.cc:299] Negotiation successful
0924 21:08:09.650752 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":675,"thread_start_us":63,"threads_started":1}
I20260924 21:08:09.650842 2707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:09.647278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42367 (local address 127.0.98.1:51553)
0924 21:08:09.648061 (+ 783us) negotiation.cc:107] Waiting for socket to connect
0924 21:08:09.648072 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 21:08:09.648127 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 21:08:09.648337 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 21:08:09.648337 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 21:08:09.648339 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 21:08:09.648393 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:09.648395 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:09.649122 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:09.649123 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:09.649964 (+ 841us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:09.649967 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:09.650583 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:09.650584 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:09.650659 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:09.650663 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 21:08:09.650730 (+ 67us) client_negotiation.cc:240] Negotiation successful
0924 21:08:09.650768 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":764,"thread_start_us":78,"threads_started":1}
I20260924 21:08:09.651063 2411 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10b3b8d3228d4b78a0e4ca23a0cf3fee" candidate_uuid: "508ed3a686554bd6af0204840904ab09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a5f5a1ce0964d918460512bd4a094d1" is_pre_election: true
I20260924 21:08:09.651135 2411 raft_consensus.cc:2454] T 10b3b8d3228d4b78a0e4ca23a0cf3fee P 7a5f5a1ce0964d918460512bd4a094d1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 508ed3a686554bd6af0204840904ab09 in term 0.
I20260924 21:08:09.651274 2203 leader_election.cc:302] T 10b3b8d3228d4b78a0e4ca23a0cf3fee P 508ed3a686554bd6af0204840904ab09 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 508ed3a686554bd6af0204840904ab09, 7a5f5a1ce0964d918460512bd4a094d1; no voters:
I20260924 21:08:09.651387 2703 raft_consensus.cc:2781] T 10b3b8d3228d4b78a0e4ca23a0cf3fee P 508ed3a686554bd6af0204840904ab09 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 21:08:09.651415 2703 raft_consensus.cc:482] T 10b3b8d3228d4b78a0e4ca23a0cf3fee P 508ed3a686554bd6af0204840904ab09 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 21:08:09.651443 2703 raft_consensus.cc:3037] T 10b3b8d3228d4b78a0e4ca23a0cf3fee P 508ed3a686554bd6af0204840904ab09 [term 0 FOLLOWER]: Advancing to term 1
I20260924 21:08:09.652060 2703 raft_consensus.cc:504] T 10b3b8d3228d4b78a0e4ca23a0cf3fee P 508ed3a686554bd6af0204840904ab09 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "508ed3a686554bd6af0204840904ab09" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44497 } } peers { permanent_uuid: "7a5f5a1ce0964d918460512bd4a094d1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42367 } } peers { permanent_uuid: "592523081f024e1a976a2d02808f79b4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33863 } }
I20260924 21:08:09.652199 2703 leader_election.cc:288] T 10b3b8d3228d4b78a0e4ca23a0cf3fee P 508ed3a686554bd6af0204840904ab09 [CANDIDATE]: Term 1 election: Requested vote from peers 7a5f5a1ce0964d918460512bd4a094d1 (127.0.98.2:42367), 592523081f024e1a976a2d02808f79b4 (127.0.98.3:33863)
I20260924 21:08:09.652271 2411 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10b3b8d3228d4b78a0e4ca23a0cf3fee" candidate_uuid: "508ed3a686554bd6af0204840904ab09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a5f5a1ce0964d918460512bd4a094d1"
I20260924 21:08:09.652324 2411 raft_consensus.cc:3037] T 10b3b8d3228d4b78a0e4ca23a0cf3fee P 7a5f5a1ce0964d918460512bd4a094d1 [term 0 FOLLOWER]: Advancing to term 1
I20260924 21:08:09.652359 2705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:09.647498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33863 (local address 127.0.98.1:38057)
0924 21:08:09.649171 (+ 1673us) negotiation.cc:107] Waiting for socket to connect
0924 21:08:09.649179 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0924 21:08:09.649223 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 21:08:09.649336 (+ 113us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 21:08:09.649337 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 21:08:09.649339 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 21:08:09.649602 (+ 263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:09.649604 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:09.650330 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:09.650331 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:09.651471 (+ 1140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:09.651474 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:09.652116 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:09.652117 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:09.652190 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:09.652194 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 21:08:09.652258 (+ 64us) client_negotiation.cc:240] Negotiation successful
0924 21:08:09.652292 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1653,"thread_start_us":61,"threads_started":1}
I20260924 21:08:09.652379 2708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:09.647615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:38057 (local address 127.0.98.3:33863)
0924 21:08:09.647742 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0924 21:08:09.647744 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 21:08:09.649213 (+ 1469us) server_negotiation.cc:407] Connection header received
0924 21:08:09.649264 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:08:09.649264 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:08:09.649273 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:08:09.649334 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 21:08:09.649673 (+ 339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:09.650222 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:09.651550 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:09.652024 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:09.652093 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:09.652104 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0924 21:08:09.652265 (+ 161us) server_negotiation.cc:299] Negotiation successful
0924 21:08:09.652309 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101,"thread_start_us":79,"threads_started":1}
I20260924 21:08:09.652616 2542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10b3b8d3228d4b78a0e4ca23a0cf3fee" candidate_uuid: "508ed3a686554bd6af0204840904ab09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "592523081f024e1a976a2d02808f79b4" is_pre_election: true
I20260924 21:08:09.652637 2541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10b3b8d3228d4b78a0e4ca23a0cf3fee" candidate_uuid: "508ed3a686554bd6af0204840904ab09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "592523081f024e1a976a2d02808f79b4"
I20260924 21:08:09.652736 2542 raft_consensus.cc:2454] T 10b3b8d3228d4b78a0e4ca23a0cf3fee P 592523081f024e1a976a2d02808f79b4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 508ed3a686554bd6af0204840904ab09 in term 0.
I20260924 21:08:09.652868 2411 raft_consensus.cc:2454] T 10b3b8d3228d4b78a0e4ca23a0cf3fee P 7a5f5a1ce0964d918460512bd4a094d1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 508ed3a686554bd6af0204840904ab09 in term 1.
I20260924 21:08:09.653012 2203 leader_election.cc:302] T 10b3b8d3228d4b78a0e4ca23a0cf3fee P 508ed3a686554bd6af0204840904ab09 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 508ed3a686554bd6af0204840904ab09, 7a5f5a1ce0964d918460512bd4a094d1; no voters: 592523081f024e1a976a2d02808f79b4
I20260924 21:08:09.653101 2703 raft_consensus.cc:2781] T 10b3b8d3228d4b78a0e4ca23a0cf3fee P 508ed3a686554bd6af0204840904ab09 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 21:08:09.653144 2703 raft_consensus.cc:686] T 10b3b8d3228d4b78a0e4ca23a0cf3fee P 508ed3a686554bd6af0204840904ab09 [term 1 LEADER]: Becoming Leader. State: Replica: 508ed3a686554bd6af0204840904ab09, State: Running, Role: LEADER
I20260924 21:08:09.653194 2703 consensus_queue.cc:237] T 10b3b8d3228d4b78a0e4ca23a0cf3fee P 508ed3a686554bd6af0204840904ab09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "508ed3a686554bd6af0204840904ab09" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44497 } } peers { permanent_uuid: "7a5f5a1ce0964d918460512bd4a094d1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42367 } } peers { permanent_uuid: "592523081f024e1a976a2d02808f79b4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33863 } }
I20260924 21:08:09.653785 2114 catalog_manager.cc:5495] T 10b3b8d3228d4b78a0e4ca23a0cf3fee P 508ed3a686554bd6af0204840904ab09 reported cstate change: term changed from 0 to 1, leader changed from <none> to 508ed3a686554bd6af0204840904ab09 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "508ed3a686554bd6af0204840904ab09" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "508ed3a686554bd6af0204840904ab09" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44497 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7a5f5a1ce0964d918460512bd4a094d1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42367 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "592523081f024e1a976a2d02808f79b4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33863 } health_report { overall_health: UNKNOWN } } }
I20260924 21:08:09.661270 2713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:09.659409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43762 (local address 127.0.98.1:44497)
0924 21:08:09.659527 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0924 21:08:09.659529 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 21:08:09.659535 (+ 6us) server_negotiation.cc:407] Connection header received
0924 21:08:09.659562 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:08:09.659562 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:08:09.659580 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:08:09.659633 (+ 53us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 21:08:09.659718 (+ 85us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:09.660102 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:09.660713 (+ 611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:09.660868 (+ 155us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:09.660910 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:09.661036 (+ 126us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 21:08:09.661113 (+ 77us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 21:08:09.661153 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0924 21:08:09.661202 (+ 49us) server_negotiation.cc:299] Negotiation successful
0924 21:08:09.661213 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103,"thread_start_us":78,"threads_started":1}
I20260924 21:08:09.665478 2708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:09.663192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:45705 (local address 127.0.98.3:33863)
0924 21:08:09.663308 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0924 21:08:09.663310 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 21:08:09.663320 (+ 10us) server_negotiation.cc:407] Connection header received
0924 21:08:09.663365 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:08:09.663366 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:08:09.663377 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:08:09.663433 (+ 56us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 21:08:09.663537 (+ 104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:09.664207 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:09.664982 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:09.665136 (+ 154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:09.665190 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:09.665305 (+ 115us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 21:08:09.665328 (+ 23us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 21:08:09.665375 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0924 21:08:09.665414 (+ 39us) server_negotiation.cc:299] Negotiation successful
0924 21:08:09.665430 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
I20260924 21:08:09.665483 2716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:09.663135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33863 (local address 127.0.98.1:45705)
0924 21:08:09.663240 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0924 21:08:09.663261 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0924 21:08:09.663307 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 21:08:09.663436 (+ 129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 21:08:09.663436 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 21:08:09.663437 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 21:08:09.663487 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:09.663488 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:09.664279 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:09.664280 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:09.664894 (+ 614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:09.664896 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:09.665197 (+ 301us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:09.665198 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:09.665235 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:09.665245 (+ 10us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 21:08:09.665382 (+ 137us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 21:08:09.665383 (+ 1us) client_negotiation.cc:769] Sending connection context
0924 21:08:09.665411 (+ 28us) client_negotiation.cc:240] Negotiation successful
0924 21:08:09.665437 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":73,"thread_start_us":48,"threads_started":1}
I20260924 21:08:09.667174 2411 raft_consensus.cc:1261] T 10b3b8d3228d4b78a0e4ca23a0cf3fee P 7a5f5a1ce0964d918460512bd4a094d1 [term 1 FOLLOWER]: Refusing update from remote peer 508ed3a686554bd6af0204840904ab09: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 21:08:09.667279 2542 raft_consensus.cc:3037] T 10b3b8d3228d4b78a0e4ca23a0cf3fee P 592523081f024e1a976a2d02808f79b4 [term 0 FOLLOWER]: Advancing to term 1
I20260924 21:08:09.667382 2704 consensus_queue.cc:1035] T 10b3b8d3228d4b78a0e4ca23a0cf3fee P 508ed3a686554bd6af0204840904ab09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a5f5a1ce0964d918460512bd4a094d1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42367 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 21:08:09.668133 2542 raft_consensus.cc:1261] T 10b3b8d3228d4b78a0e4ca23a0cf3fee P 592523081f024e1a976a2d02808f79b4 [term 1 FOLLOWER]: Refusing update from remote peer 508ed3a686554bd6af0204840904ab09: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 21:08:09.668394 2704 consensus_queue.cc:1035] T 10b3b8d3228d4b78a0e4ca23a0cf3fee P 508ed3a686554bd6af0204840904ab09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "592523081f024e1a976a2d02808f79b4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 21:08:09.674458 2713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:09.671996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:52725 (local address 127.0.98.1:44497)
0924 21:08:09.672046 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0924 21:08:09.672047 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 21:08:09.672175 (+ 128us) server_negotiation.cc:407] Connection header received
0924 21:08:09.672230 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:08:09.672230 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:08:09.672237 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:08:09.672280 (+ 43us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 21:08:09.672403 (+ 123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:09.672812 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:09.673770 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:09.674016 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:09.674096 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:09.674242 (+ 146us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 21:08:09.674271 (+ 29us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 21:08:09.674322 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0924 21:08:09.674383 (+ 61us) server_negotiation.cc:299] Negotiation successful
0924 21:08:09.674400 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260924 21:08:09.674489 2722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:09.671944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44497 (local address 127.0.98.3:52725)
0924 21:08:09.672101 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0924 21:08:09.672125 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0924 21:08:09.672178 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 21:08:09.672313 (+ 135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 21:08:09.672313 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 21:08:09.672315 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 21:08:09.672351 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:09.672352 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:09.672899 (+ 547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:09.672900 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:09.673629 (+ 729us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:09.673631 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:09.674098 (+ 467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:09.674099 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:09.674152 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:09.674156 (+ 4us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 21:08:09.674324 (+ 168us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 21:08:09.674326 (+ 2us) client_negotiation.cc:769] Sending connection context
0924 21:08:09.674375 (+ 49us) client_negotiation.cc:240] Negotiation successful
0924 21:08:09.674416 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":66,"threads_started":1}
I20260924 21:08:09.680163 2114 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57966:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 21:08:09.680255 2114 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57966:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 21:08:09.680805 2114 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 224fefe284af480497048ba6af0e88cf: Sending DeleteTablet for 3 replicas of tablet 10b3b8d3228d4b78a0e4ca23a0cf3fee
I20260924 21:08:09.681020 2251 tablet_service.cc:1514] Processing DeleteTablet for tablet 10b3b8d3228d4b78a0e4ca23a0cf3fee with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 21:08:09 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:43754
I20260924 21:08:09.681046 2531 tablet_service.cc:1514] Processing DeleteTablet for tablet 10b3b8d3228d4b78a0e4ca23a0cf3fee with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 21:08:09 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:53752
I20260924 21:08:09.681092 2689 tablet_replica.cc:330] T 10b3b8d3228d4b78a0e4ca23a0cf3fee P 508ed3a686554bd6af0204840904ab09: stopping tablet replica
I20260924 21:08:09.681108 2688 tablet_replica.cc:330] T 10b3b8d3228d4b78a0e4ca23a0cf3fee P 592523081f024e1a976a2d02808f79b4: stopping tablet replica
I20260924 21:08:09.681129 2689 raft_consensus.cc:2229] T 10b3b8d3228d4b78a0e4ca23a0cf3fee P 508ed3a686554bd6af0204840904ab09 [term 1 LEADER]: Raft consensus shutting down.
I20260924 21:08:09.681143 2688 raft_consensus.cc:2229] T 10b3b8d3228d4b78a0e4ca23a0cf3fee P 592523081f024e1a976a2d02808f79b4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 21:08:09.681169 2688 raft_consensus.cc:2258] T 10b3b8d3228d4b78a0e4ca23a0cf3fee P 592523081f024e1a976a2d02808f79b4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 21:08:09.681185 2689 raft_consensus.cc:2258] T 10b3b8d3228d4b78a0e4ca23a0cf3fee P 508ed3a686554bd6af0204840904ab09 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 21:08:09.681377 2391 tablet_service.cc:1514] Processing DeleteTablet for tablet 10b3b8d3228d4b78a0e4ca23a0cf3fee with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 21:08:09 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:45502
I20260924 21:08:09.681484 2689 ts_tablet_manager.cc:1905] T 10b3b8d3228d4b78a0e4ca23a0cf3fee P 508ed3a686554bd6af0204840904ab09: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 21:08:09.681509 2687 tablet_replica.cc:330] T 10b3b8d3228d4b78a0e4ca23a0cf3fee P 7a5f5a1ce0964d918460512bd4a094d1: stopping tablet replica
I20260924 21:08:09.681551 2687 raft_consensus.cc:2229] T 10b3b8d3228d4b78a0e4ca23a0cf3fee P 7a5f5a1ce0964d918460512bd4a094d1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 21:08:09.681581 2687 raft_consensus.cc:2258] T 10b3b8d3228d4b78a0e4ca23a0cf3fee P 7a5f5a1ce0964d918460512bd4a094d1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 21:08:09.681845 2688 ts_tablet_manager.cc:1905] T 10b3b8d3228d4b78a0e4ca23a0cf3fee P 592523081f024e1a976a2d02808f79b4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 21:08:09.681989 2114 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57966:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 21:08:09.682111 2114 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 21:08:09.682909 2689 ts_tablet_manager.cc:1918] T 10b3b8d3228d4b78a0e4ca23a0cf3fee P 508ed3a686554bd6af0204840904ab09: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 21:08:09.682958 2689 log.cc:1196] T 10b3b8d3228d4b78a0e4ca23a0cf3fee P 508ed3a686554bd6af0204840904ab09: Deleting WAL directory at /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790284076800039-392-0/minicluster-data/ts-0/wal/wals/10b3b8d3228d4b78a0e4ca23a0cf3fee
I20260924 21:08:09.683140 2689 ts_tablet_manager.cc:1939] T 10b3b8d3228d4b78a0e4ca23a0cf3fee P 508ed3a686554bd6af0204840904ab09: Deleting consensus metadata
I20260924 21:08:09.683224 2688 ts_tablet_manager.cc:1918] T 10b3b8d3228d4b78a0e4ca23a0cf3fee P 592523081f024e1a976a2d02808f79b4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 21:08:09.683269 2688 log.cc:1196] T 10b3b8d3228d4b78a0e4ca23a0cf3fee P 592523081f024e1a976a2d02808f79b4: Deleting WAL directory at /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790284076800039-392-0/minicluster-data/ts-2/wal/wals/10b3b8d3228d4b78a0e4ca23a0cf3fee
I20260924 21:08:09.683385 2101 catalog_manager.cc:4841] TS 508ed3a686554bd6af0204840904ab09 (127.0.98.1:44497): tablet 10b3b8d3228d4b78a0e4ca23a0cf3fee (table test-table [id=9c9328f2dfc0491db35053e14c573b31]) successfully deleted
I20260924 21:08:09.683439 2688 ts_tablet_manager.cc:1939] T 10b3b8d3228d4b78a0e4ca23a0cf3fee P 592523081f024e1a976a2d02808f79b4: Deleting consensus metadata
I20260924 21:08:09.683732 2687 ts_tablet_manager.cc:1905] T 10b3b8d3228d4b78a0e4ca23a0cf3fee P 7a5f5a1ce0964d918460512bd4a094d1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 21:08:09.683776 2391 tablet_service.cc:1467] Processing CreateTablet for tablet de0ddc023ad94e1db37e690042f891d3 (DEFAULT_TABLE table=test-table [id=45f2fbbef7754c239f670d66447b9a85]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 21:08:09.683841 2531 tablet_service.cc:1467] Processing CreateTablet for tablet de0ddc023ad94e1db37e690042f891d3 (DEFAULT_TABLE table=test-table [id=45f2fbbef7754c239f670d66447b9a85]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 21:08:09.683879 2391 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet de0ddc023ad94e1db37e690042f891d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 21:08:09.683911 2531 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet de0ddc023ad94e1db37e690042f891d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 21:08:09.685055 2702 tablet_bootstrap.cc:492] T de0ddc023ad94e1db37e690042f891d3 P 7a5f5a1ce0964d918460512bd4a094d1: Bootstrap starting.
I20260924 21:08:09.685097 2699 tablet_bootstrap.cc:492] T de0ddc023ad94e1db37e690042f891d3 P 592523081f024e1a976a2d02808f79b4: Bootstrap starting.
I20260924 21:08:09.685159 2251 tablet_service.cc:1467] Processing CreateTablet for tablet de0ddc023ad94e1db37e690042f891d3 (DEFAULT_TABLE table=test-table [id=45f2fbbef7754c239f670d66447b9a85]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 21:08:09.685259 2251 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet de0ddc023ad94e1db37e690042f891d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 21:08:09.685338 2699 tablet_bootstrap.cc:654] T de0ddc023ad94e1db37e690042f891d3 P 592523081f024e1a976a2d02808f79b4: Neither blocks nor log segments found. Creating new log.
I20260924 21:08:09.685443 2702 tablet_bootstrap.cc:654] T de0ddc023ad94e1db37e690042f891d3 P 7a5f5a1ce0964d918460512bd4a094d1: Neither blocks nor log segments found. Creating new log.
I20260924 21:08:09.685756 2098 catalog_manager.cc:4841] TS 592523081f024e1a976a2d02808f79b4 (127.0.98.3:33863): tablet 10b3b8d3228d4b78a0e4ca23a0cf3fee (table test-table [id=9c9328f2dfc0491db35053e14c573b31]) successfully deleted
I20260924 21:08:09.685060 2687 ts_tablet_manager.cc:1918] T 10b3b8d3228d4b78a0e4ca23a0cf3fee P 7a5f5a1ce0964d918460512bd4a094d1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 21:08:09.686015 2687 log.cc:1196] T 10b3b8d3228d4b78a0e4ca23a0cf3fee P 7a5f5a1ce0964d918460512bd4a094d1: Deleting WAL directory at /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790284076800039-392-0/minicluster-data/ts-1/wal/wals/10b3b8d3228d4b78a0e4ca23a0cf3fee
I20260924 21:08:09.686105 2699 tablet_bootstrap.cc:492] T de0ddc023ad94e1db37e690042f891d3 P 592523081f024e1a976a2d02808f79b4: No bootstrap required, opened a new log
I20260924 21:08:09.686142 2699 ts_tablet_manager.cc:1397] T de0ddc023ad94e1db37e690042f891d3 P 592523081f024e1a976a2d02808f79b4: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260924 21:08:09.686220 2687 ts_tablet_manager.cc:1939] T 10b3b8d3228d4b78a0e4ca23a0cf3fee P 7a5f5a1ce0964d918460512bd4a094d1: Deleting consensus metadata
I20260924 21:08:09.686251 2699 raft_consensus.cc:348] T de0ddc023ad94e1db37e690042f891d3 P 592523081f024e1a976a2d02808f79b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "508ed3a686554bd6af0204840904ab09" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44497 } } peers { permanent_uuid: "7a5f5a1ce0964d918460512bd4a094d1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42367 } } peers { permanent_uuid: "592523081f024e1a976a2d02808f79b4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33863 } }
I20260924 21:08:09.686303 2699 raft_consensus.cc:374] T de0ddc023ad94e1db37e690042f891d3 P 592523081f024e1a976a2d02808f79b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 21:08:09.686323 2699 raft_consensus.cc:729] T de0ddc023ad94e1db37e690042f891d3 P 592523081f024e1a976a2d02808f79b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 592523081f024e1a976a2d02808f79b4, State: Initialized, Role: FOLLOWER
I20260924 21:08:09.686358 2699 consensus_queue.cc:260] T de0ddc023ad94e1db37e690042f891d3 P 592523081f024e1a976a2d02808f79b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "508ed3a686554bd6af0204840904ab09" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44497 } } peers { permanent_uuid: "7a5f5a1ce0964d918460512bd4a094d1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42367 } } peers { permanent_uuid: "592523081f024e1a976a2d02808f79b4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33863 } }
I20260924 21:08:09.686431 2699 ts_tablet_manager.cc:1428] T de0ddc023ad94e1db37e690042f891d3 P 592523081f024e1a976a2d02808f79b4: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 21:08:09.686465 2698 tablet_bootstrap.cc:492] T de0ddc023ad94e1db37e690042f891d3 P 508ed3a686554bd6af0204840904ab09: Bootstrap starting.
I20260924 21:08:09.686492 2099 catalog_manager.cc:4841] TS 7a5f5a1ce0964d918460512bd4a094d1 (127.0.98.2:42367): tablet 10b3b8d3228d4b78a0e4ca23a0cf3fee (table test-table [id=9c9328f2dfc0491db35053e14c573b31]) successfully deleted
I20260924 21:08:09.686851 2698 tablet_bootstrap.cc:654] T de0ddc023ad94e1db37e690042f891d3 P 508ed3a686554bd6af0204840904ab09: Neither blocks nor log segments found. Creating new log.
I20260924 21:08:09.687186 2702 tablet_bootstrap.cc:492] T de0ddc023ad94e1db37e690042f891d3 P 7a5f5a1ce0964d918460512bd4a094d1: No bootstrap required, opened a new log
I20260924 21:08:09.687239 2702 ts_tablet_manager.cc:1397] T de0ddc023ad94e1db37e690042f891d3 P 7a5f5a1ce0964d918460512bd4a094d1: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 21:08:09.687363 2702 raft_consensus.cc:348] T de0ddc023ad94e1db37e690042f891d3 P 7a5f5a1ce0964d918460512bd4a094d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "508ed3a686554bd6af0204840904ab09" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44497 } } peers { permanent_uuid: "7a5f5a1ce0964d918460512bd4a094d1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42367 } } peers { permanent_uuid: "592523081f024e1a976a2d02808f79b4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33863 } }
I20260924 21:08:09.687398 2698 tablet_bootstrap.cc:492] T de0ddc023ad94e1db37e690042f891d3 P 508ed3a686554bd6af0204840904ab09: No bootstrap required, opened a new log
I20260924 21:08:09.687439 2702 raft_consensus.cc:374] T de0ddc023ad94e1db37e690042f891d3 P 7a5f5a1ce0964d918460512bd4a094d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 21:08:09.687449 2698 ts_tablet_manager.cc:1397] T de0ddc023ad94e1db37e690042f891d3 P 508ed3a686554bd6af0204840904ab09: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260924 21:08:09.687465 2702 raft_consensus.cc:729] T de0ddc023ad94e1db37e690042f891d3 P 7a5f5a1ce0964d918460512bd4a094d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a5f5a1ce0964d918460512bd4a094d1, State: Initialized, Role: FOLLOWER
I20260924 21:08:09.687510 2702 consensus_queue.cc:260] T de0ddc023ad94e1db37e690042f891d3 P 7a5f5a1ce0964d918460512bd4a094d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "508ed3a686554bd6af0204840904ab09" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44497 } } peers { permanent_uuid: "7a5f5a1ce0964d918460512bd4a094d1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42367 } } peers { permanent_uuid: "592523081f024e1a976a2d02808f79b4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33863 } }
I20260924 21:08:09.687608 2702 ts_tablet_manager.cc:1428] T de0ddc023ad94e1db37e690042f891d3 P 7a5f5a1ce0964d918460512bd4a094d1: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 21:08:09.687588 2698 raft_consensus.cc:348] T de0ddc023ad94e1db37e690042f891d3 P 508ed3a686554bd6af0204840904ab09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "508ed3a686554bd6af0204840904ab09" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44497 } } peers { permanent_uuid: "7a5f5a1ce0964d918460512bd4a094d1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42367 } } peers { permanent_uuid: "592523081f024e1a976a2d02808f79b4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33863 } }
I20260924 21:08:09.687646 2698 raft_consensus.cc:374] T de0ddc023ad94e1db37e690042f891d3 P 508ed3a686554bd6af0204840904ab09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 21:08:09.687667 2698 raft_consensus.cc:729] T de0ddc023ad94e1db37e690042f891d3 P 508ed3a686554bd6af0204840904ab09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 508ed3a686554bd6af0204840904ab09, State: Initialized, Role: FOLLOWER
I20260924 21:08:09.687722 2698 consensus_queue.cc:260] T de0ddc023ad94e1db37e690042f891d3 P 508ed3a686554bd6af0204840904ab09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "508ed3a686554bd6af0204840904ab09" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44497 } } peers { permanent_uuid: "7a5f5a1ce0964d918460512bd4a094d1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42367 } } peers { permanent_uuid: "592523081f024e1a976a2d02808f79b4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33863 } }
I20260924 21:08:09.687826 2698 ts_tablet_manager.cc:1428] T de0ddc023ad94e1db37e690042f891d3 P 508ed3a686554bd6af0204840904ab09: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 21:08:09.886292 2720 raft_consensus.cc:482] T de0ddc023ad94e1db37e690042f891d3 P 508ed3a686554bd6af0204840904ab09 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 21:08:09.886371 2720 raft_consensus.cc:504] T de0ddc023ad94e1db37e690042f891d3 P 508ed3a686554bd6af0204840904ab09 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "508ed3a686554bd6af0204840904ab09" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44497 } } peers { permanent_uuid: "7a5f5a1ce0964d918460512bd4a094d1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42367 } } peers { permanent_uuid: "592523081f024e1a976a2d02808f79b4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33863 } }
I20260924 21:08:09.886547 2720 leader_election.cc:288] T de0ddc023ad94e1db37e690042f891d3 P 508ed3a686554bd6af0204840904ab09 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7a5f5a1ce0964d918460512bd4a094d1 (127.0.98.2:42367), 592523081f024e1a976a2d02808f79b4 (127.0.98.3:33863)
I20260924 21:08:09.886852 2411 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de0ddc023ad94e1db37e690042f891d3" candidate_uuid: "508ed3a686554bd6af0204840904ab09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a5f5a1ce0964d918460512bd4a094d1" is_pre_election: true
I20260924 21:08:09.886878 2542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de0ddc023ad94e1db37e690042f891d3" candidate_uuid: "508ed3a686554bd6af0204840904ab09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "592523081f024e1a976a2d02808f79b4" is_pre_election: true
I20260924 21:08:09.886989 2411 raft_consensus.cc:2454] T de0ddc023ad94e1db37e690042f891d3 P 7a5f5a1ce0964d918460512bd4a094d1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 508ed3a686554bd6af0204840904ab09 in term 0.
I20260924 21:08:09.886993 2542 raft_consensus.cc:2454] T de0ddc023ad94e1db37e690042f891d3 P 592523081f024e1a976a2d02808f79b4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 508ed3a686554bd6af0204840904ab09 in term 0.
I20260924 21:08:09.887190 2202 leader_election.cc:302] T de0ddc023ad94e1db37e690042f891d3 P 508ed3a686554bd6af0204840904ab09 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 508ed3a686554bd6af0204840904ab09, 592523081f024e1a976a2d02808f79b4; no voters:
I20260924 21:08:09.887297 2720 raft_consensus.cc:2781] T de0ddc023ad94e1db37e690042f891d3 P 508ed3a686554bd6af0204840904ab09 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 21:08:09.887346 2720 raft_consensus.cc:482] T de0ddc023ad94e1db37e690042f891d3 P 508ed3a686554bd6af0204840904ab09 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 21:08:09.887377 2720 raft_consensus.cc:3037] T de0ddc023ad94e1db37e690042f891d3 P 508ed3a686554bd6af0204840904ab09 [term 0 FOLLOWER]: Advancing to term 1
I20260924 21:08:09.888056 2720 raft_consensus.cc:504] T de0ddc023ad94e1db37e690042f891d3 P 508ed3a686554bd6af0204840904ab09 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "508ed3a686554bd6af0204840904ab09" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44497 } } peers { permanent_uuid: "7a5f5a1ce0964d918460512bd4a094d1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42367 } } peers { permanent_uuid: "592523081f024e1a976a2d02808f79b4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33863 } }
I20260924 21:08:09.888193 2720 leader_election.cc:288] T de0ddc023ad94e1db37e690042f891d3 P 508ed3a686554bd6af0204840904ab09 [CANDIDATE]: Term 1 election: Requested vote from peers 7a5f5a1ce0964d918460512bd4a094d1 (127.0.98.2:42367), 592523081f024e1a976a2d02808f79b4 (127.0.98.3:33863)
I20260924 21:08:09.888329 2542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de0ddc023ad94e1db37e690042f891d3" candidate_uuid: "508ed3a686554bd6af0204840904ab09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "592523081f024e1a976a2d02808f79b4"
I20260924 21:08:09.888350 2411 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de0ddc023ad94e1db37e690042f891d3" candidate_uuid: "508ed3a686554bd6af0204840904ab09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a5f5a1ce0964d918460512bd4a094d1"
I20260924 21:08:09.888428 2542 raft_consensus.cc:3037] T de0ddc023ad94e1db37e690042f891d3 P 592523081f024e1a976a2d02808f79b4 [term 0 FOLLOWER]: Advancing to term 1
I20260924 21:08:09.888424 2411 raft_consensus.cc:3037] T de0ddc023ad94e1db37e690042f891d3 P 7a5f5a1ce0964d918460512bd4a094d1 [term 0 FOLLOWER]: Advancing to term 1
I20260924 21:08:09.889056 2411 raft_consensus.cc:2454] T de0ddc023ad94e1db37e690042f891d3 P 7a5f5a1ce0964d918460512bd4a094d1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 508ed3a686554bd6af0204840904ab09 in term 1.
I20260924 21:08:09.889169 2542 raft_consensus.cc:2454] T de0ddc023ad94e1db37e690042f891d3 P 592523081f024e1a976a2d02808f79b4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 508ed3a686554bd6af0204840904ab09 in term 1.
I20260924 21:08:09.889201 2203 leader_election.cc:302] T de0ddc023ad94e1db37e690042f891d3 P 508ed3a686554bd6af0204840904ab09 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 508ed3a686554bd6af0204840904ab09, 7a5f5a1ce0964d918460512bd4a094d1; no voters:
I20260924 21:08:09.889297 2720 raft_consensus.cc:2781] T de0ddc023ad94e1db37e690042f891d3 P 508ed3a686554bd6af0204840904ab09 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 21:08:09.889349 2720 raft_consensus.cc:686] T de0ddc023ad94e1db37e690042f891d3 P 508ed3a686554bd6af0204840904ab09 [term 1 LEADER]: Becoming Leader. State: Replica: 508ed3a686554bd6af0204840904ab09, State: Running, Role: LEADER
I20260924 21:08:09.889389 2720 consensus_queue.cc:237] T de0ddc023ad94e1db37e690042f891d3 P 508ed3a686554bd6af0204840904ab09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "508ed3a686554bd6af0204840904ab09" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44497 } } peers { permanent_uuid: "7a5f5a1ce0964d918460512bd4a094d1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42367 } } peers { permanent_uuid: "592523081f024e1a976a2d02808f79b4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33863 } }
I20260924 21:08:09.890013 2113 catalog_manager.cc:5495] T de0ddc023ad94e1db37e690042f891d3 P 508ed3a686554bd6af0204840904ab09 reported cstate change: term changed from 0 to 1, leader changed from <none> to 508ed3a686554bd6af0204840904ab09 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "508ed3a686554bd6af0204840904ab09" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "508ed3a686554bd6af0204840904ab09" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44497 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7a5f5a1ce0964d918460512bd4a094d1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42367 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "592523081f024e1a976a2d02808f79b4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33863 } health_report { overall_health: UNKNOWN } } }
I20260924 21:08:09.914482 2411 raft_consensus.cc:1261] T de0ddc023ad94e1db37e690042f891d3 P 7a5f5a1ce0964d918460512bd4a094d1 [term 1 FOLLOWER]: Refusing update from remote peer 508ed3a686554bd6af0204840904ab09: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 21:08:09.914494 2542 raft_consensus.cc:1261] T de0ddc023ad94e1db37e690042f891d3 P 592523081f024e1a976a2d02808f79b4 [term 1 FOLLOWER]: Refusing update from remote peer 508ed3a686554bd6af0204840904ab09: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 21:08:09.914662 2720 consensus_queue.cc:1035] T de0ddc023ad94e1db37e690042f891d3 P 508ed3a686554bd6af0204840904ab09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a5f5a1ce0964d918460512bd4a094d1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42367 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 21:08:09.914748 2703 consensus_queue.cc:1035] T de0ddc023ad94e1db37e690042f891d3 P 508ed3a686554bd6af0204840904ab09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "592523081f024e1a976a2d02808f79b4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260924 21:08:09.918413 2671 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260924 21:08:11.434628 2113 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57966:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 21:08:11.434722 2113 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57966:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 21:08:11.435369 2113 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 224fefe284af480497048ba6af0e88cf: Sending DeleteTablet for 3 replicas of tablet de0ddc023ad94e1db37e690042f891d3
I20260924 21:08:11.435637 2251 tablet_service.cc:1514] Processing DeleteTablet for tablet de0ddc023ad94e1db37e690042f891d3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 21:08:11 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:43754
I20260924 21:08:11.435647 2391 tablet_service.cc:1514] Processing DeleteTablet for tablet de0ddc023ad94e1db37e690042f891d3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 21:08:11 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:45502
I20260924 21:08:11.435735 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAzNNjS/build/release/bin/kudu with pid 2186
I20260924 21:08:11.436229 2531 tablet_service.cc:1514] Processing DeleteTablet for tablet de0ddc023ad94e1db37e690042f891d3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 21:08:11 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:53752
I20260924 21:08:11.436342 2753 tablet_replica.cc:330] T de0ddc023ad94e1db37e690042f891d3 P 7a5f5a1ce0964d918460512bd4a094d1: stopping tablet replica
I20260924 21:08:11.436399 2754 tablet_replica.cc:330] T de0ddc023ad94e1db37e690042f891d3 P 592523081f024e1a976a2d02808f79b4: stopping tablet replica
I20260924 21:08:11.436436 2753 raft_consensus.cc:2229] T de0ddc023ad94e1db37e690042f891d3 P 7a5f5a1ce0964d918460512bd4a094d1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 21:08:11.436453 2754 raft_consensus.cc:2229] T de0ddc023ad94e1db37e690042f891d3 P 592523081f024e1a976a2d02808f79b4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 21:08:11.436486 2754 raft_consensus.cc:2258] T de0ddc023ad94e1db37e690042f891d3 P 592523081f024e1a976a2d02808f79b4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 21:08:11.436502 2753 raft_consensus.cc:2258] T de0ddc023ad94e1db37e690042f891d3 P 7a5f5a1ce0964d918460512bd4a094d1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 21:08:11.436726 2754 ts_tablet_manager.cc:1905] T de0ddc023ad94e1db37e690042f891d3 P 592523081f024e1a976a2d02808f79b4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 21:08:11.436729 2753 ts_tablet_manager.cc:1905] T de0ddc023ad94e1db37e690042f891d3 P 7a5f5a1ce0964d918460512bd4a094d1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 21:08:11.438261 2754 ts_tablet_manager.cc:1918] T de0ddc023ad94e1db37e690042f891d3 P 592523081f024e1a976a2d02808f79b4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260924 21:08:11.438331 2754 log.cc:1196] T de0ddc023ad94e1db37e690042f891d3 P 592523081f024e1a976a2d02808f79b4: Deleting WAL directory at /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790284076800039-392-0/minicluster-data/ts-2/wal/wals/de0ddc023ad94e1db37e690042f891d3
I20260924 21:08:11.438555 2754 ts_tablet_manager.cc:1939] T de0ddc023ad94e1db37e690042f891d3 P 592523081f024e1a976a2d02808f79b4: Deleting consensus metadata
I20260924 21:08:11.438683 2753 ts_tablet_manager.cc:1918] T de0ddc023ad94e1db37e690042f891d3 P 7a5f5a1ce0964d918460512bd4a094d1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260924 21:08:11.438748 2753 log.cc:1196] T de0ddc023ad94e1db37e690042f891d3 P 7a5f5a1ce0964d918460512bd4a094d1: Deleting WAL directory at /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790284076800039-392-0/minicluster-data/ts-1/wal/wals/de0ddc023ad94e1db37e690042f891d3
I20260924 21:08:11.438889 2098 catalog_manager.cc:4841] TS 592523081f024e1a976a2d02808f79b4 (127.0.98.3:33863): tablet de0ddc023ad94e1db37e690042f891d3 (table test-table [id=45f2fbbef7754c239f670d66447b9a85]) successfully deleted
I20260924 21:08:11.438971 2753 ts_tablet_manager.cc:1939] T de0ddc023ad94e1db37e690042f891d3 P 7a5f5a1ce0964d918460512bd4a094d1: Deleting consensus metadata
I20260924 21:08:11.439539 2099 catalog_manager.cc:4841] TS 7a5f5a1ce0964d918460512bd4a094d1 (127.0.98.2:42367): tablet de0ddc023ad94e1db37e690042f891d3 (table test-table [id=45f2fbbef7754c239f670d66447b9a85]) successfully deleted
W20260924 21:08:11.443192 2101 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:44497 (error 108)
I20260924 21:08:11.443485 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAzNNjS/build/release/bin/kudu with pid 2326
I20260924 21:08:11.444041 2756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:11.443576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44497 (local address 127.0.0.1:43768)
0924 21:08:11.443964 (+ 388us) negotiation.cc:107] Waiting for socket to connect
0924 21:08:11.444005 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:44497: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":368,"thread_start_us":53,"threads_started":1}
W20260924 21:08:11.444131 2101 catalog_manager.cc:4584] TS 508ed3a686554bd6af0204840904ab09 (127.0.98.1:44497): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet de0ddc023ad94e1db37e690042f891d3: Network error: Client connection negotiation failed: client connection to 127.0.98.1:44497: connect: Connection refused (error 111)
I20260924 21:08:11.447989 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAzNNjS/build/release/bin/kudu with pid 2466
I20260924 21:08:11.452883 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAzNNjS/build/release/bin/kudu with pid 2082
2026-09-24T21:08:11Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (5722 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790284076800039-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 21:08:11 dist-test-slave-c7tk krb5kdc[2759](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 21:08:11 dist-test-slave-c7tk krb5kdc[2759](info): set up 2 sockets
Sep 24 21:08:11 dist-test-slave-c7tk krb5kdc[2759](info): commencing operation
krb5kdc: starting...
W20260924 21:08:12.377471 392 mini_kdc.cc:121] Time spent starting KDC: real 0.905s user 0.000s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790284076800039-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790284076800039-392-0/krb5kdc/test-user.keytab.
Sep 24 21:08:12 dist-test-slave-c7tk krb5kdc[2759](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284092, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T21:08:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T21:08:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790284076800039-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790284076800039-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 21:08:12.480396 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAzNNjS/build/release/bin/kudu
/tmp/dist-test-taskAzNNjS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790284076800039-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790284076800039-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790284076800039-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790284076800039-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39799
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790284076800039-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37455
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39799
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790284076800039-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790284076800039-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790284076800039-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 21:08:12.549084 2772 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 21:08:12.549243 2772 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 21:08:12.549263 2772 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 21:08:12.550348 2772 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 21:08:12.550384 2772 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 21:08:12.550397 2772 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 21:08:12.550410 2772 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 21:08:12.550451 2772 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 21:08:12.551771 2772 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37455
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790284076800039-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790284076800039-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39799
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790284076800039-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39799
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790284076800039-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2772
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790284076800039-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 19:49:06 UTC on ccycloud.quasar-vhqjoc.root.comops.site
build id 4670864
I20260924 21:08:12.551987 2772 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 21:08:12.552179 2772 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 21:08:12.556632 2778 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 21:08:12.556600 2780 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 21:08:12.556618 2777 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 21:08:12.556655 2772 server_base.cc:1034] running on GCE node
I20260924 21:08:12.557035 2772 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 21:08:12.557301 2772 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 21:08:12.558439 2772 hybrid_clock.cc:648] HybridClock initialized: now 1790284092558441 us; error 39 us; skew 500 ppm
Sep 24 21:08:12 dist-test-slave-c7tk krb5kdc[2759](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284092, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 21:08:12.564863 2772 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 21:08:12.565256 2772 webserver.cc:460] Webserver started at http://127.0.98.62:46613/ using document root <none> and password file <none>
I20260924 21:08:12.565481 2772 fs_manager.cc:359] Metadata directory not provided
I20260924 21:08:12.565536 2772 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 21:08:12.565656 2772 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 21:08:12.566645 2772 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790284076800039-392-0/minicluster-data/master-0/data/instance:
uuid: "95968188e2054886b00afd74801761f6"
format_stamp: "Formatted at 2026-09-24 21:08:12 on dist-test-slave-c7tk"
server_key: "65f6ae9c182dcff8c786c3ad123e13a3"
server_key_iv: "488cd9f50559f83c88466d2efc82a0ae"
server_key_version: "encryptionkey@0"
I20260924 21:08:12.566951 2772 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790284076800039-392-0/minicluster-data/master-0/wal/instance:
uuid: "95968188e2054886b00afd74801761f6"
format_stamp: "Formatted at 2026-09-24 21:08:12 on dist-test-slave-c7tk"
server_key: "65f6ae9c182dcff8c786c3ad123e13a3"
server_key_iv: "488cd9f50559f83c88466d2efc82a0ae"
server_key_version: "encryptionkey@0"
I20260924 21:08:12.568224 2772 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 21:08:12.568873 2787 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 21:08:12.569022 2772 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 21:08:12.569108 2772 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790284076800039-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790284076800039-392-0/minicluster-data/master-0/wal
uuid: "95968188e2054886b00afd74801761f6"
format_stamp: "Formatted at 2026-09-24 21:08:12 on dist-test-slave-c7tk"
server_key: "65f6ae9c182dcff8c786c3ad123e13a3"
server_key_iv: "488cd9f50559f83c88466d2efc82a0ae"
server_key_version: "encryptionkey@0"
I20260924 21:08:12.569178 2772 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790284076800039-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790284076800039-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790284076800039-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 21:08:12.585007 2772 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 21:08:12.586656 2772 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 21:08:12.586818 2772 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 21:08:12.590353 2772 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39799
I20260924 21:08:12.590375 2839 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39799 every 8 connection(s)
I20260924 21:08:12.590833 2772 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790284076800039-392-0/minicluster-data/master-0/data/info.pb
I20260924 21:08:12.591382 2840 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 21:08:12.593127 2840 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 95968188e2054886b00afd74801761f6: Bootstrap starting.
I20260924 21:08:12.593554 2840 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 95968188e2054886b00afd74801761f6: Neither blocks nor log segments found. Creating new log.
I20260924 21:08:12.593789 2840 log.cc:824] T 00000000000000000000000000000000 P 95968188e2054886b00afd74801761f6: Log is configured to *not* fsync() on all Append() calls
I20260924 21:08:12.593981 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAzNNjS/build/release/bin/kudu as pid 2772
I20260924 21:08:12.594075 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790284076800039-392-0/minicluster-data/master-0/wal/instance
I20260924 21:08:12.594244 392 external_mini_cluster.cc:1428] Setting key 4fdc84b63207e5d2edace98738143989
I20260924 21:08:12.594533 2840 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 95968188e2054886b00afd74801761f6: No bootstrap required, opened a new log
I20260924 21:08:12.596834 2840 raft_consensus.cc:348] T 00000000000000000000000000000000 P 95968188e2054886b00afd74801761f6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95968188e2054886b00afd74801761f6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39799 } }
I20260924 21:08:12.596962 2840 raft_consensus.cc:374] T 00000000000000000000000000000000 P 95968188e2054886b00afd74801761f6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 21:08:12.597000 2840 raft_consensus.cc:729] T 00000000000000000000000000000000 P 95968188e2054886b00afd74801761f6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 95968188e2054886b00afd74801761f6, State: Initialized, Role: FOLLOWER
I20260924 21:08:12.597100 2840 consensus_queue.cc:260] T 00000000000000000000000000000000 P 95968188e2054886b00afd74801761f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95968188e2054886b00afd74801761f6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39799 } }
I20260924 21:08:12.597163 2840 raft_consensus.cc:388] T 00000000000000000000000000000000 P 95968188e2054886b00afd74801761f6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 21:08:12.597198 2840 raft_consensus.cc:482] T 00000000000000000000000000000000 P 95968188e2054886b00afd74801761f6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 21:08:12.597239 2840 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 95968188e2054886b00afd74801761f6 [term 0 FOLLOWER]: Advancing to term 1
I20260924 21:08:12.597725 2840 raft_consensus.cc:504] T 00000000000000000000000000000000 P 95968188e2054886b00afd74801761f6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95968188e2054886b00afd74801761f6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39799 } }
I20260924 21:08:12.597848 2840 leader_election.cc:302] T 00000000000000000000000000000000 P 95968188e2054886b00afd74801761f6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 95968188e2054886b00afd74801761f6; no voters:
I20260924 21:08:12.598011 2840 leader_election.cc:288] T 00000000000000000000000000000000 P 95968188e2054886b00afd74801761f6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 21:08:12.598050 2845 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 95968188e2054886b00afd74801761f6 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 21:08:12.598191 2845 raft_consensus.cc:686] T 00000000000000000000000000000000 P 95968188e2054886b00afd74801761f6 [term 1 LEADER]: Becoming Leader. State: Replica: 95968188e2054886b00afd74801761f6, State: Running, Role: LEADER
I20260924 21:08:12.598224 2840 sys_catalog.cc:564] T 00000000000000000000000000000000 P 95968188e2054886b00afd74801761f6 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 21:08:12.598347 2845 consensus_queue.cc:237] T 00000000000000000000000000000000 P 95968188e2054886b00afd74801761f6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95968188e2054886b00afd74801761f6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39799 } }
I20260924 21:08:12.598721 2846 sys_catalog.cc:455] T 00000000000000000000000000000000 P 95968188e2054886b00afd74801761f6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "95968188e2054886b00afd74801761f6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95968188e2054886b00afd74801761f6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39799 } } }
I20260924 21:08:12.598795 2846 sys_catalog.cc:458] T 00000000000000000000000000000000 P 95968188e2054886b00afd74801761f6 [sys.catalog]: This master's current role is: LEADER
I20260924 21:08:12.598932 2847 sys_catalog.cc:455] T 00000000000000000000000000000000 P 95968188e2054886b00afd74801761f6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 95968188e2054886b00afd74801761f6. Latest consensus state: current_term: 1 leader_uuid: "95968188e2054886b00afd74801761f6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95968188e2054886b00afd74801761f6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39799 } } }
I20260924 21:08:12.599023 2847 sys_catalog.cc:458] T 00000000000000000000000000000000 P 95968188e2054886b00afd74801761f6 [sys.catalog]: This master's current role is: LEADER
W20260924 21:08:12.600111 2861 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 95968188e2054886b00afd74801761f6: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 21:08:12.600162 2861 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 21:08:12.600221 2852 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 21:08:12.600356 2852 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 24 21:08:12 dist-test-slave-c7tk krb5kdc[2759](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284092, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 21:08:12.601482 2852 catalog_manager.cc:1347] Generated new cluster ID: 44927c5e68e0409f97f96828c7d927ec
I20260924 21:08:12.601528 2852 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 21:08:12.604550 2844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:12.594639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48094 (local address 127.0.98.62:39799)
0924 21:08:12.594783 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0924 21:08:12.594803 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0924 21:08:12.594829 (+ 26us) server_negotiation.cc:407] Connection header received
0924 21:08:12.595561 (+ 732us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:08:12.595567 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:08:12.595605 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:08:12.595696 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:08:12.596348 (+ 652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:12.597192 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:12.598016 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:12.598608 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:12.598718 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:12.601192 (+ 2474us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 21:08:12.601195 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 21:08:12.601196 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 21:08:12.601205 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 21:08:12.602913 (+ 1708us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:08:12.603263 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:08:12.603265 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:08:12.603266 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:08:12.603351 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:08:12.603530 (+ 179us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:08:12.603531 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:08:12.603531 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:08:12.603764 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 21:08:12.603904 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0924 21:08:12.604268 (+ 364us) server_negotiation.cc:299] Negotiation successful
0924 21:08:12.604380 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102,"thread_start_us":79,"threads_started":1}
I20260924 21:08:12.609810 2852 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 21:08:12.610256 2852 catalog_manager.cc:1504] Loading token signing keys...
I20260924 21:08:12.615091 2852 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 95968188e2054886b00afd74801761f6: Generated new TSK 0
I20260924 21:08:12.615227 2852 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790284076800039-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790284076800039-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260924 21:08:12.644325 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAzNNjS/build/release/bin/kudu
/tmp/dist-test-taskAzNNjS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790284076800039-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790284076800039-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790284076800039-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790284076800039-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39799
--keytab_file=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790284076800039-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37455
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790284076800039-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790284076800039-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790284076800039-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 21:08:12.726836 2866 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 21:08:12.726997 2866 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 21:08:12.727016 2866 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 21:08:12.728111 2866 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 21:08:12.728152 2866 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 21:08:12.728176 2866 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 21:08:12.729499 2866 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37455
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790284076800039-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790284076800039-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790284076800039-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790284076800039-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39799
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2866
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790284076800039-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 19:49:06 UTC on ccycloud.quasar-vhqjoc.root.comops.site
build id 4670864
I20260924 21:08:12.729688 2866 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 21:08:12.729890 2866 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 21:08:12.733973 2866 server_base.cc:1034] running on GCE node
W20260924 21:08:12.733992 2874 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 21:08:12.734107 2871 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 21:08:12.734063 2872 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 21:08:12.734287 2866 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 21:08:12.734493 2866 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 21:08:12.735639 2866 hybrid_clock.cc:648] HybridClock initialized: now 1790284092735623 us; error 27 us; skew 500 ppm
Sep 24 21:08:12 dist-test-slave-c7tk krb5kdc[2759](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284092, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 21:08:12.737763 2866 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 21:08:12.738178 2866 webserver.cc:460] Webserver started at http://127.0.98.1:42753/ using document root <none> and password file <none>
I20260924 21:08:12.738376 2866 fs_manager.cc:359] Metadata directory not provided
I20260924 21:08:12.738437 2866 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 21:08:12.738591 2866 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 21:08:12.739521 2866 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790284076800039-392-0/minicluster-data/ts-0/data/instance:
uuid: "a0f8841e14844e4182c25c0fe700f304"
format_stamp: "Formatted at 2026-09-24 21:08:12 on dist-test-slave-c7tk"
server_key: "59b3f119555434831dd78453c5e83659"
server_key_iv: "c2e74c2801b7cfd63e37e09e73ab4d86"
server_key_version: "encryptionkey@0"
I20260924 21:08:12.739847 2866 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790284076800039-392-0/minicluster-data/ts-0/wal/instance:
uuid: "a0f8841e14844e4182c25c0fe700f304"
format_stamp: "Formatted at 2026-09-24 21:08:12 on dist-test-slave-c7tk"
server_key: "59b3f119555434831dd78453c5e83659"
server_key_iv: "c2e74c2801b7cfd63e37e09e73ab4d86"
server_key_version: "encryptionkey@0"
I20260924 21:08:12.741077 2866 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 21:08:12.741768 2881 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 21:08:12.741958 2866 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 21:08:12.742053 2866 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790284076800039-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790284076800039-392-0/minicluster-data/ts-0/wal
uuid: "a0f8841e14844e4182c25c0fe700f304"
format_stamp: "Formatted at 2026-09-24 21:08:12 on dist-test-slave-c7tk"
server_key: "59b3f119555434831dd78453c5e83659"
server_key_iv: "c2e74c2801b7cfd63e37e09e73ab4d86"
server_key_version: "encryptionkey@0"
I20260924 21:08:12.742122 2866 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790284076800039-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790284076800039-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790284076800039-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 21:08:12.757485 2866 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 21:08:12.759483 2866 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 21:08:12.759634 2866 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 21:08:12.759817 2866 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 21:08:12.760072 2866 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 21:08:12.760118 2866 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 21:08:12.760154 2866 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 21:08:12.760176 2866 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 21:08:12.765118 2866 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:37417
I20260924 21:08:12.765151 2994 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:37417 every 8 connection(s)
I20260924 21:08:12.765409 2866 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790284076800039-392-0/minicluster-data/ts-0/data/info.pb
I20260924 21:08:12.769090 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAzNNjS/build/release/bin/kudu as pid 2866
I20260924 21:08:12.769179 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790284076800039-392-0/minicluster-data/ts-0/wal/instance
I20260924 21:08:12.769363 392 external_mini_cluster.cc:1428] Setting key 7399db337f7e1ea937fdae79efc21c73
Sep 24 21:08:12 dist-test-slave-c7tk krb5kdc[2759](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284092, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 21:08:12.773388 2844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:12.765997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:56691 (local address 127.0.98.62:39799)
0924 21:08:12.766048 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0924 21:08:12.766050 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 21:08:12.766207 (+ 157us) server_negotiation.cc:407] Connection header received
0924 21:08:12.766935 (+ 728us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:08:12.766936 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:08:12.766944 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:08:12.766980 (+ 36us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:08:12.767447 (+ 467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:12.767926 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:12.768795 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:12.768957 (+ 162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:12.769003 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:12.770944 (+ 1941us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 21:08:12.770945 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 21:08:12.770946 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 21:08:12.770952 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 21:08:12.772060 (+ 1108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:08:12.772536 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:08:12.772537 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:08:12.772537 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:08:12.772587 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:08:12.772825 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:08:12.772825 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:08:12.772826 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:08:12.772941 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 21:08:12.773007 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0924 21:08:12.773235 (+ 228us) server_negotiation.cc:299] Negotiation successful
0924 21:08:12.773310 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260924 21:08:12.773538 2997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:12.765981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39799 (local address 127.0.98.1:56691)
0924 21:08:12.766110 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0924 21:08:12.766141 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0924 21:08:12.766851 (+ 710us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 21:08:12.767021 (+ 170us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 21:08:12.767024 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 21:08:12.767207 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0924 21:08:12.767357 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:12.767359 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:12.767994 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:12.767994 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:12.768731 (+ 737us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:12.768732 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:12.769023 (+ 291us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:12.769024 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:12.769157 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:12.769582 (+ 425us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 21:08:12.769591 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 21:08:12.770847 (+ 1256us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 21:08:12.772159 (+ 1312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 21:08:12.772164 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 21:08:12.772166 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 21:08:12.772451 (+ 285us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 21:08:12.772649 (+ 198us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 21:08:12.772650 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 21:08:12.772651 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 21:08:12.772748 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 21:08:12.773004 (+ 256us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 21:08:12.773010 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 21:08:12.773127 (+ 117us) client_negotiation.cc:769] Sending connection context
0924 21:08:12.773217 (+ 90us) client_negotiation.cc:240] Negotiation successful
0924 21:08:12.773341 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":71,"threads_started":1}
I20260924 21:08:12.774034 2995 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39799
I20260924 21:08:12.774138 2995 heartbeater.cc:464] Registering TS with master...
I20260924 21:08:12.774307 2995 heartbeater.cc:511] Master 127.0.98.62:39799 requested a full tablet report, sending...
I20260924 21:08:12.774720 2804 ts_manager.cc:194] Registered new tserver with Master: a0f8841e14844e4182c25c0fe700f304 (127.0.98.1:37417)
I20260924 21:08:12.775240 2804 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:56691
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790284076800039-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790284076800039-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260924 21:08:12.790933 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAzNNjS/build/release/bin/kudu
/tmp/dist-test-taskAzNNjS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790284076800039-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790284076800039-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790284076800039-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790284076800039-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39799
--keytab_file=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790284076800039-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37455
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790284076800039-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790284076800039-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790284076800039-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 21:08:12.860841 3000 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 21:08:12.861007 3000 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 21:08:12.861027 3000 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 21:08:12.861985 3000 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 21:08:12.862015 3000 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 21:08:12.862035 3000 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 21:08:12.863194 3000 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37455
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790284076800039-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790284076800039-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790284076800039-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790284076800039-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39799
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3000
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790284076800039-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 19:49:06 UTC on ccycloud.quasar-vhqjoc.root.comops.site
build id 4670864
I20260924 21:08:12.863334 3000 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 21:08:12.863473 3000 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 21:08:12.867828 3005 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 21:08:12.867901 3008 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 21:08:12.867928 3006 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 21:08:12.868044 3000 server_base.cc:1034] running on GCE node
I20260924 21:08:12.868250 3000 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 21:08:12.868455 3000 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 21:08:12.869568 3000 hybrid_clock.cc:648] HybridClock initialized: now 1790284092869561 us; error 23 us; skew 500 ppm
Sep 24 21:08:12 dist-test-slave-c7tk krb5kdc[2759](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284092, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 21:08:12.871790 3000 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 21:08:12.872171 3000 webserver.cc:460] Webserver started at http://127.0.98.2:33363/ using document root <none> and password file <none>
I20260924 21:08:12.872357 3000 fs_manager.cc:359] Metadata directory not provided
I20260924 21:08:12.872394 3000 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 21:08:12.872550 3000 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 21:08:12.873411 3000 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790284076800039-392-0/minicluster-data/ts-1/data/instance:
uuid: "ec96c27762534fe29e3ca0e2e89c0452"
format_stamp: "Formatted at 2026-09-24 21:08:12 on dist-test-slave-c7tk"
server_key: "55adcde07776eff72256f008d347ee3f"
server_key_iv: "9a1e038754a9132ee3197ab8d94574ed"
server_key_version: "encryptionkey@0"
I20260924 21:08:12.873762 3000 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790284076800039-392-0/minicluster-data/ts-1/wal/instance:
uuid: "ec96c27762534fe29e3ca0e2e89c0452"
format_stamp: "Formatted at 2026-09-24 21:08:12 on dist-test-slave-c7tk"
server_key: "55adcde07776eff72256f008d347ee3f"
server_key_iv: "9a1e038754a9132ee3197ab8d94574ed"
server_key_version: "encryptionkey@0"
I20260924 21:08:12.874823 3000 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 21:08:12.875329 3015 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 21:08:12.875448 3000 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 21:08:12.875530 3000 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790284076800039-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790284076800039-392-0/minicluster-data/ts-1/wal
uuid: "ec96c27762534fe29e3ca0e2e89c0452"
format_stamp: "Formatted at 2026-09-24 21:08:12 on dist-test-slave-c7tk"
server_key: "55adcde07776eff72256f008d347ee3f"
server_key_iv: "9a1e038754a9132ee3197ab8d94574ed"
server_key_version: "encryptionkey@0"
I20260924 21:08:12.875594 3000 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790284076800039-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790284076800039-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790284076800039-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 21:08:12.891480 3000 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 21:08:12.893270 3000 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 21:08:12.893411 3000 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 21:08:12.893611 3000 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 21:08:12.893927 3000 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 21:08:12.893972 3000 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 21:08:12.894003 3000 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 21:08:12.894023 3000 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 21:08:12.899171 3000 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:35115
I20260924 21:08:12.899209 3128 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:35115 every 8 connection(s)
I20260924 21:08:12.899452 3000 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790284076800039-392-0/minicluster-data/ts-1/data/info.pb
I20260924 21:08:12.904277 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAzNNjS/build/release/bin/kudu as pid 3000
I20260924 21:08:12.904368 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790284076800039-392-0/minicluster-data/ts-1/wal/instance
I20260924 21:08:12.904557 392 external_mini_cluster.cc:1428] Setting key 7f87e7ca5d5cc5dd087cda22f96dc415
Sep 24 21:08:12 dist-test-slave-c7tk krb5kdc[2759](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284092, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 21:08:12.907851 2844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:12.900052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:45445 (local address 127.0.98.62:39799)
0924 21:08:12.900144 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0924 21:08:12.900146 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 21:08:12.900314 (+ 168us) server_negotiation.cc:407] Connection header received
0924 21:08:12.901144 (+ 830us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:08:12.901145 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:08:12.901157 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:08:12.901200 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:08:12.901721 (+ 521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:12.902206 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:12.903034 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:12.903235 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:12.903305 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:12.905197 (+ 1892us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 21:08:12.905198 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 21:08:12.905199 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 21:08:12.905205 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 21:08:12.906526 (+ 1321us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:08:12.906966 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:08:12.906967 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:08:12.906967 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:08:12.907012 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:08:12.907255 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:08:12.907256 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:08:12.907256 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:08:12.907337 (+ 81us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 21:08:12.907380 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0924 21:08:12.907691 (+ 311us) server_negotiation.cc:299] Negotiation successful
0924 21:08:12.907766 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260924 21:08:12.908006 3131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:12.900034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39799 (local address 127.0.98.2:45445)
0924 21:08:12.900203 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0924 21:08:12.900242 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0924 21:08:12.901046 (+ 804us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 21:08:12.901282 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 21:08:12.901285 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 21:08:12.901502 (+ 217us) client_negotiation.cc:189] Negotiated authn=SASL
0924 21:08:12.901643 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:12.901646 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:12.902280 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:12.902281 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:12.902947 (+ 666us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:12.902948 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:12.903313 (+ 365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:12.903314 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:12.903428 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:12.903760 (+ 332us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 21:08:12.903767 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0924 21:08:12.905119 (+ 1352us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 21:08:12.906609 (+ 1490us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 21:08:12.906615 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 21:08:12.906617 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 21:08:12.906897 (+ 280us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 21:08:12.907070 (+ 173us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 21:08:12.907071 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 21:08:12.907071 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 21:08:12.907194 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 21:08:12.907389 (+ 195us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 21:08:12.907418 (+ 29us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 21:08:12.907564 (+ 146us) client_negotiation.cc:769] Sending connection context
0924 21:08:12.907664 (+ 100us) client_negotiation.cc:240] Negotiation successful
0924 21:08:12.907799 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":106,"threads_started":1}
I20260924 21:08:12.908401 3129 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39799
I20260924 21:08:12.908483 3129 heartbeater.cc:464] Registering TS with master...
I20260924 21:08:12.908659 3129 heartbeater.cc:511] Master 127.0.98.62:39799 requested a full tablet report, sending...
I20260924 21:08:12.908954 2804 ts_manager.cc:194] Registered new tserver with Master: ec96c27762534fe29e3ca0e2e89c0452 (127.0.98.2:35115)
I20260924 21:08:12.909417 2804 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:45445
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790284076800039-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790284076800039-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260924 21:08:12.927783 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAzNNjS/build/release/bin/kudu
/tmp/dist-test-taskAzNNjS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790284076800039-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790284076800039-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790284076800039-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790284076800039-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39799
--keytab_file=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790284076800039-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37455
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790284076800039-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790284076800039-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790284076800039-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 21:08:13.000505 3134 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 21:08:13.000695 3134 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 21:08:13.000728 3134 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 21:08:13.001683 3134 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 21:08:13.001740 3134 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 21:08:13.001780 3134 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 21:08:13.003100 3134 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37455
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790284076800039-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790284076800039-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790284076800039-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790284076800039-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39799
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3134
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790284076800039-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 19:49:06 UTC on ccycloud.quasar-vhqjoc.root.comops.site
build id 4670864
I20260924 21:08:13.003338 3134 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 21:08:13.003561 3134 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 21:08:13.007997 3142 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 21:08:13.008065 3134 server_base.cc:1034] running on GCE node
W20260924 21:08:13.008066 3140 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 21:08:13.007999 3139 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 21:08:13.008425 3134 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 21:08:13.008677 3134 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 21:08:13.009835 3134 hybrid_clock.cc:648] HybridClock initialized: now 1790284093009831 us; error 36 us; skew 500 ppm
Sep 24 21:08:13 dist-test-slave-c7tk krb5kdc[2759](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284093, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 21:08:13.011862 3134 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 21:08:13.012166 3134 webserver.cc:460] Webserver started at http://127.0.98.3:38387/ using document root <none> and password file <none>
I20260924 21:08:13.012355 3134 fs_manager.cc:359] Metadata directory not provided
I20260924 21:08:13.012408 3134 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 21:08:13.012518 3134 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 21:08:13.013329 3134 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790284076800039-392-0/minicluster-data/ts-2/data/instance:
uuid: "6fb5754b717c4677b3f912e00f4bfe9c"
format_stamp: "Formatted at 2026-09-24 21:08:13 on dist-test-slave-c7tk"
server_key: "b4cceb6e985d52b7e3d85fc69185d8d7"
server_key_iv: "0048743af79fdcd873c6d05ce143ff89"
server_key_version: "encryptionkey@0"
I20260924 21:08:13.013633 3134 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790284076800039-392-0/minicluster-data/ts-2/wal/instance:
uuid: "6fb5754b717c4677b3f912e00f4bfe9c"
format_stamp: "Formatted at 2026-09-24 21:08:13 on dist-test-slave-c7tk"
server_key: "b4cceb6e985d52b7e3d85fc69185d8d7"
server_key_iv: "0048743af79fdcd873c6d05ce143ff89"
server_key_version: "encryptionkey@0"
I20260924 21:08:13.014972 3134 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 21:08:13.015564 3149 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 21:08:13.015718 3134 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 21:08:13.015795 3134 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790284076800039-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790284076800039-392-0/minicluster-data/ts-2/wal
uuid: "6fb5754b717c4677b3f912e00f4bfe9c"
format_stamp: "Formatted at 2026-09-24 21:08:13 on dist-test-slave-c7tk"
server_key: "b4cceb6e985d52b7e3d85fc69185d8d7"
server_key_iv: "0048743af79fdcd873c6d05ce143ff89"
server_key_version: "encryptionkey@0"
I20260924 21:08:13.015861 3134 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790284076800039-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790284076800039-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790284076800039-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 21:08:13.028512 3134 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 21:08:13.029976 3134 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 21:08:13.030097 3134 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 21:08:13.030272 3134 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 21:08:13.030506 3134 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 21:08:13.030550 3134 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 21:08:13.030580 3134 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 21:08:13.030606 3134 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 21:08:13.035826 3134 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:44361
I20260924 21:08:13.035879 3262 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:44361 every 8 connection(s)
I20260924 21:08:13.036099 3134 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790284076800039-392-0/minicluster-data/ts-2/data/info.pb
Sep 24 21:08:13 dist-test-slave-c7tk krb5kdc[2759](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284093, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 21:08:13.041464 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAzNNjS/build/release/bin/kudu as pid 3134
I20260924 21:08:13.041551 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790284076800039-392-0/minicluster-data/ts-2/wal/instance
I20260924 21:08:13.041760 392 external_mini_cluster.cc:1428] Setting key 9ee6c144b277789dc9f275ecbbaff2fd
I20260924 21:08:13.043999 2844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:13.036676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:57989 (local address 127.0.98.62:39799)
0924 21:08:13.036727 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0924 21:08:13.036729 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 21:08:13.036891 (+ 162us) server_negotiation.cc:407] Connection header received
0924 21:08:13.037728 (+ 837us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:08:13.037729 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:08:13.037742 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:08:13.037787 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:08:13.038343 (+ 556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:13.038819 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:13.039725 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:13.039918 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:13.039980 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:13.041723 (+ 1743us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 21:08:13.041724 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 21:08:13.041724 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 21:08:13.041730 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 21:08:13.043024 (+ 1294us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:08:13.043379 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:08:13.043379 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:08:13.043380 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:08:13.043438 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:08:13.043592 (+ 154us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:08:13.043593 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:08:13.043593 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:08:13.043680 (+ 87us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 21:08:13.043725 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0924 21:08:13.043892 (+ 167us) server_negotiation.cc:299] Negotiation successful
0924 21:08:13.043944 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260924 21:08:13.044132 3265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:13.036656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39799 (local address 127.0.98.3:57989)
0924 21:08:13.036781 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0924 21:08:13.036817 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0924 21:08:13.037621 (+ 804us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 21:08:13.037871 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 21:08:13.037874 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 21:08:13.038100 (+ 226us) client_negotiation.cc:189] Negotiated authn=SASL
0924 21:08:13.038256 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:13.038259 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:13.038907 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:13.038908 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:13.039617 (+ 709us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:13.039618 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:13.039984 (+ 366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:13.039985 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:13.040087 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:13.040420 (+ 333us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 21:08:13.040427 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0924 21:08:13.041643 (+ 1216us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 21:08:13.043088 (+ 1445us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 21:08:13.043092 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 21:08:13.043094 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 21:08:13.043321 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 21:08:13.043476 (+ 155us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 21:08:13.043477 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 21:08:13.043477 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 21:08:13.043548 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 21:08:13.043717 (+ 169us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 21:08:13.043718 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 21:08:13.043804 (+ 86us) client_negotiation.cc:769] Sending connection context
0924 21:08:13.043870 (+ 66us) client_negotiation.cc:240] Negotiation successful
0924 21:08:13.043968 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":91,"thread_start_us":58,"threads_started":1}
I20260924 21:08:13.044550 3263 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39799
I20260924 21:08:13.044631 3263 heartbeater.cc:464] Registering TS with master...
I20260924 21:08:13.044761 3263 heartbeater.cc:511] Master 127.0.98.62:39799 requested a full tablet report, sending...
I20260924 21:08:13.045054 2804 ts_manager.cc:194] Registered new tserver with Master: 6fb5754b717c4677b3f912e00f4bfe9c (127.0.98.3:44361)
I20260924 21:08:13.045370 2804 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:57989
I20260924 21:08:13.053267 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260924 21:08:13.062184 2844 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 21:08:13.061465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48106 (local address 127.0.98.62:39799)
0924 21:08:13.061510 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0924 21:08:13.061511 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 21:08:13.061605 (+ 94us) server_negotiation.cc:407] Connection header received
0924 21:08:13.061639 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:08:13.061640 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:08:13.061647 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:08:13.061690 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:08:13.062121 (+ 431us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48106: BlockingRecv error: recv got EOF from 127.0.0.1:48106 (error 108)
Metrics: {"server-negotiator.queue_time_us":28}
I20260924 21:08:13.063053 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAzNNjS/build/release/bin/kudu with pid 2866
I20260924 21:08:13.067121 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAzNNjS/build/release/bin/kudu with pid 3000
I20260924 21:08:13.071139 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAzNNjS/build/release/bin/kudu with pid 3134
I20260924 21:08:13.074656 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAzNNjS/build/release/bin/kudu with pid 2772
2026-09-24T21:08:13Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (1620 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790284076800039-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 21:08:13 dist-test-slave-c7tk krb5kdc[3275](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 21:08:13 dist-test-slave-c7tk krb5kdc[3275](info): set up 2 sockets
Sep 24 21:08:13 dist-test-slave-c7tk krb5kdc[3275](info): commencing operation
krb5kdc: starting...
W20260924 21:08:13.997061 392 mini_kdc.cc:121] Time spent starting KDC: real 0.904s user 0.004s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790284076800039-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790284076800039-392-0/krb5kdc/test-user.keytab.
Sep 24 21:08:14 dist-test-slave-c7tk krb5kdc[3275](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284094, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T21:08:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T21:08:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790284076800039-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790284076800039-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 21:08:14.098294 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAzNNjS/build/release/bin/kudu
/tmp/dist-test-taskAzNNjS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790284076800039-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790284076800039-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790284076800039-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790284076800039-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41729
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790284076800039-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35979
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41729
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790284076800039-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790284076800039-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790284076800039-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 21:08:14.167815 3288 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 21:08:14.168022 3288 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 21:08:14.168061 3288 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 21:08:14.169116 3288 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 21:08:14.169164 3288 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 21:08:14.169185 3288 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 21:08:14.169210 3288 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 21:08:14.169230 3288 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 21:08:14.170466 3288 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35979
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790284076800039-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790284076800039-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41729
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790284076800039-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41729
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790284076800039-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3288
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790284076800039-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 19:49:06 UTC on ccycloud.quasar-vhqjoc.root.comops.site
build id 4670864
I20260924 21:08:14.170679 3288 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 21:08:14.170852 3288 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 21:08:14.174842 3288 server_base.cc:1034] running on GCE node
W20260924 21:08:14.174883 3294 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 21:08:14.175017 3296 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 21:08:14.174865 3293 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 21:08:14.175328 3288 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 21:08:14.175529 3288 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 21:08:14.176669 3288 hybrid_clock.cc:648] HybridClock initialized: now 1790284094176658 us; error 25 us; skew 500 ppm
Sep 24 21:08:14 dist-test-slave-c7tk krb5kdc[3275](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284094, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 21:08:14.178894 3288 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 21:08:14.179221 3288 webserver.cc:460] Webserver started at http://127.0.98.62:46035/ using document root <none> and password file <none>
I20260924 21:08:14.179409 3288 fs_manager.cc:359] Metadata directory not provided
I20260924 21:08:14.179468 3288 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 21:08:14.179593 3288 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 21:08:14.180437 3288 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790284076800039-392-0/minicluster-data/master-0/data/instance:
uuid: "0437a79f3bf44ef0a70ff419eb84b8b4"
format_stamp: "Formatted at 2026-09-24 21:08:14 on dist-test-slave-c7tk"
server_key: "438fbb6e07bf051a0cc5654160f8a6d7"
server_key_iv: "5b24120cd8b21fba496a0bde26b54b8c"
server_key_version: "encryptionkey@0"
I20260924 21:08:14.180771 3288 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790284076800039-392-0/minicluster-data/master-0/wal/instance:
uuid: "0437a79f3bf44ef0a70ff419eb84b8b4"
format_stamp: "Formatted at 2026-09-24 21:08:14 on dist-test-slave-c7tk"
server_key: "438fbb6e07bf051a0cc5654160f8a6d7"
server_key_iv: "5b24120cd8b21fba496a0bde26b54b8c"
server_key_version: "encryptionkey@0"
I20260924 21:08:14.182013 3288 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 21:08:14.182695 3303 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 21:08:14.182844 3288 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 21:08:14.182935 3288 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790284076800039-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790284076800039-392-0/minicluster-data/master-0/wal
uuid: "0437a79f3bf44ef0a70ff419eb84b8b4"
format_stamp: "Formatted at 2026-09-24 21:08:14 on dist-test-slave-c7tk"
server_key: "438fbb6e07bf051a0cc5654160f8a6d7"
server_key_iv: "5b24120cd8b21fba496a0bde26b54b8c"
server_key_version: "encryptionkey@0"
I20260924 21:08:14.183008 3288 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790284076800039-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790284076800039-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790284076800039-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 21:08:14.208252 3288 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 21:08:14.209699 3288 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 21:08:14.209862 3288 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 21:08:14.213227 3288 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41729
I20260924 21:08:14.213254 3355 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41729 every 8 connection(s)
I20260924 21:08:14.213554 3288 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790284076800039-392-0/minicluster-data/master-0/data/info.pb
I20260924 21:08:14.214354 3356 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 21:08:14.216223 3356 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0437a79f3bf44ef0a70ff419eb84b8b4: Bootstrap starting.
I20260924 21:08:14.216715 3356 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0437a79f3bf44ef0a70ff419eb84b8b4: Neither blocks nor log segments found. Creating new log.
I20260924 21:08:14.216980 3356 log.cc:824] T 00000000000000000000000000000000 P 0437a79f3bf44ef0a70ff419eb84b8b4: Log is configured to *not* fsync() on all Append() calls
I20260924 21:08:14.217602 3356 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0437a79f3bf44ef0a70ff419eb84b8b4: No bootstrap required, opened a new log
I20260924 21:08:14.219071 3356 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0437a79f3bf44ef0a70ff419eb84b8b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0437a79f3bf44ef0a70ff419eb84b8b4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41729 } }
I20260924 21:08:14.219187 3356 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0437a79f3bf44ef0a70ff419eb84b8b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 21:08:14.219228 3356 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0437a79f3bf44ef0a70ff419eb84b8b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0437a79f3bf44ef0a70ff419eb84b8b4, State: Initialized, Role: FOLLOWER
I20260924 21:08:14.219364 3356 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0437a79f3bf44ef0a70ff419eb84b8b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0437a79f3bf44ef0a70ff419eb84b8b4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41729 } }
I20260924 21:08:14.219421 3356 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0437a79f3bf44ef0a70ff419eb84b8b4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 21:08:14.219457 3356 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0437a79f3bf44ef0a70ff419eb84b8b4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 21:08:14.219503 3356 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0437a79f3bf44ef0a70ff419eb84b8b4 [term 0 FOLLOWER]: Advancing to term 1
I20260924 21:08:14.220178 3356 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0437a79f3bf44ef0a70ff419eb84b8b4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0437a79f3bf44ef0a70ff419eb84b8b4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41729 } }
I20260924 21:08:14.220274 3356 leader_election.cc:302] T 00000000000000000000000000000000 P 0437a79f3bf44ef0a70ff419eb84b8b4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0437a79f3bf44ef0a70ff419eb84b8b4; no voters:
I20260924 21:08:14.220433 3356 leader_election.cc:288] T 00000000000000000000000000000000 P 0437a79f3bf44ef0a70ff419eb84b8b4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 21:08:14.220467 3359 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0437a79f3bf44ef0a70ff419eb84b8b4 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 21:08:14.220608 3359 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0437a79f3bf44ef0a70ff419eb84b8b4 [term 1 LEADER]: Becoming Leader. State: Replica: 0437a79f3bf44ef0a70ff419eb84b8b4, State: Running, Role: LEADER
I20260924 21:08:14.220690 3356 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0437a79f3bf44ef0a70ff419eb84b8b4 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 21:08:14.220705 3359 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0437a79f3bf44ef0a70ff419eb84b8b4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0437a79f3bf44ef0a70ff419eb84b8b4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41729 } }
I20260924 21:08:14.221050 3361 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0437a79f3bf44ef0a70ff419eb84b8b4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0437a79f3bf44ef0a70ff419eb84b8b4. Latest consensus state: current_term: 1 leader_uuid: "0437a79f3bf44ef0a70ff419eb84b8b4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0437a79f3bf44ef0a70ff419eb84b8b4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41729 } } }
I20260924 21:08:14.221135 3361 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0437a79f3bf44ef0a70ff419eb84b8b4 [sys.catalog]: This master's current role is: LEADER
I20260924 21:08:14.221055 3360 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0437a79f3bf44ef0a70ff419eb84b8b4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0437a79f3bf44ef0a70ff419eb84b8b4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0437a79f3bf44ef0a70ff419eb84b8b4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41729 } } }
I20260924 21:08:14.221261 3360 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0437a79f3bf44ef0a70ff419eb84b8b4 [sys.catalog]: This master's current role is: LEADER
I20260924 21:08:14.221336 3367 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 21:08:14.221777 3367 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 21:08:14.221985 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAzNNjS/build/release/bin/kudu as pid 3288
I20260924 21:08:14.222065 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790284076800039-392-0/minicluster-data/master-0/wal/instance
I20260924 21:08:14.222237 392 external_mini_cluster.cc:1428] Setting key 69a591442d952f3026ef4f6b4ad28cfd
I20260924 21:08:14.223263 3367 catalog_manager.cc:1347] Generated new cluster ID: 715feae07659469dacfb6e7f3cf2fc60
I20260924 21:08:14.223311 3367 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 24 21:08:14 dist-test-slave-c7tk krb5kdc[3275](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284094, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 21:08:14.230444 3378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:14.222567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34990 (local address 127.0.98.62:41729)
0924 21:08:14.222703 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0924 21:08:14.222710 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 21:08:14.222879 (+ 169us) server_negotiation.cc:407] Connection header received
0924 21:08:14.222995 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:08:14.223001 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:08:14.223048 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:08:14.223167 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:08:14.223758 (+ 591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:14.224564 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:14.225250 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:14.225643 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:14.225738 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:14.227635 (+ 1897us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 21:08:14.227639 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 21:08:14.227639 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 21:08:14.227650 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 21:08:14.229135 (+ 1485us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:08:14.229467 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:08:14.229469 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:08:14.229470 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:08:14.229541 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:08:14.229708 (+ 167us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:08:14.229709 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:08:14.229709 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:08:14.229950 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 21:08:14.230093 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0924 21:08:14.230147 (+ 54us) server_negotiation.cc:299] Negotiation successful
0924 21:08:14.230261 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102,"thread_start_us":67,"threads_started":1}
I20260924 21:08:14.231122 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 21:08:14.231460 3367 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 21:08:14.232035 3367 catalog_manager.cc:1504] Loading token signing keys...
I20260924 21:08:14.236729 3367 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0437a79f3bf44ef0a70ff419eb84b8b4: Generated new TSK 0
I20260924 21:08:14.236850 3367 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 21:08:14.244582 3378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:14.241639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35002 (local address 127.0.98.62:41729)
0924 21:08:14.241693 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0924 21:08:14.241695 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 21:08:14.241809 (+ 114us) server_negotiation.cc:407] Connection header received
0924 21:08:14.241872 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:08:14.241873 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:08:14.241880 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:08:14.241949 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:08:14.242246 (+ 297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:14.242651 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:14.243440 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:14.243619 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:14.243689 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:14.244210 (+ 521us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 21:08:14.244211 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 21:08:14.244212 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 21:08:14.244360 (+ 148us) server_negotiation.cc:922] calling sasl_server_start()
0924 21:08:14.244370 (+ 10us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 21:08:14.244375 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 21:08:14.244428 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0924 21:08:14.244502 (+ 74us) server_negotiation.cc:299] Negotiation successful
0924 21:08:14.244520 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260924 21:08:14.246534 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAzNNjS/build/release/bin/kudu with pid 3288
2026-09-24T21:08:14Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1170 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790284076800039-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 21:08:14 dist-test-slave-c7tk krb5kdc[3389](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 21:08:14 dist-test-slave-c7tk krb5kdc[3389](info): set up 2 sockets
Sep 24 21:08:14 dist-test-slave-c7tk krb5kdc[3389](info): commencing operation
krb5kdc: starting...
W20260924 21:08:15.165028 392 mini_kdc.cc:121] Time spent starting KDC: real 0.901s user 0.001s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790284076800039-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790284076800039-392-0/krb5kdc/test-user.keytab.
Sep 24 21:08:15 dist-test-slave-c7tk krb5kdc[3389](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284095, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T21:08:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T21:08:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790284076800039-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790284076800039-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 21:08:15.263602 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAzNNjS/build/release/bin/kudu
/tmp/dist-test-taskAzNNjS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790284076800039-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790284076800039-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790284076800039-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790284076800039-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45889
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790284076800039-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33643
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45889
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790284076800039-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790284076800039-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790284076800039-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 21:08:15.334483 3402 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 21:08:15.334656 3402 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 21:08:15.334676 3402 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 21:08:15.335700 3402 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 21:08:15.335734 3402 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 21:08:15.335747 3402 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 21:08:15.335790 3402 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 21:08:15.335810 3402 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 21:08:15.337155 3402 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33643
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790284076800039-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790284076800039-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45889
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790284076800039-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45889
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790284076800039-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3402
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790284076800039-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 19:49:06 UTC on ccycloud.quasar-vhqjoc.root.comops.site
build id 4670864
I20260924 21:08:15.337392 3402 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 21:08:15.337595 3402 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 21:08:15.342075 3410 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 21:08:15.342082 3408 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 21:08:15.342082 3407 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 21:08:15.342325 3402 server_base.cc:1034] running on GCE node
I20260924 21:08:15.342540 3402 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 21:08:15.342794 3402 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 21:08:15.343953 3402 hybrid_clock.cc:648] HybridClock initialized: now 1790284095343946 us; error 32 us; skew 500 ppm
Sep 24 21:08:15 dist-test-slave-c7tk krb5kdc[3389](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284095, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 21:08:15.353156 3402 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 21:08:15.353523 3402 webserver.cc:460] Webserver started at http://127.0.98.62:41721/ using document root <none> and password file <none>
I20260924 21:08:15.353756 3402 fs_manager.cc:359] Metadata directory not provided
I20260924 21:08:15.353837 3402 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 21:08:15.353978 3402 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 21:08:15.354918 3402 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790284076800039-392-0/minicluster-data/master-0/data/instance:
uuid: "679f0a58106744c4a6db809ddfe48384"
format_stamp: "Formatted at 2026-09-24 21:08:15 on dist-test-slave-c7tk"
server_key: "06fa411d0dfd2be3a3eff7f4b1a03433"
server_key_iv: "28c1e6e10c19ba26f008c885f5e6192d"
server_key_version: "encryptionkey@0"
I20260924 21:08:15.355208 3402 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790284076800039-392-0/minicluster-data/master-0/wal/instance:
uuid: "679f0a58106744c4a6db809ddfe48384"
format_stamp: "Formatted at 2026-09-24 21:08:15 on dist-test-slave-c7tk"
server_key: "06fa411d0dfd2be3a3eff7f4b1a03433"
server_key_iv: "28c1e6e10c19ba26f008c885f5e6192d"
server_key_version: "encryptionkey@0"
I20260924 21:08:15.356251 3402 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 21:08:15.356868 3417 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 21:08:15.357038 3402 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 21:08:15.357121 3402 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790284076800039-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790284076800039-392-0/minicluster-data/master-0/wal
uuid: "679f0a58106744c4a6db809ddfe48384"
format_stamp: "Formatted at 2026-09-24 21:08:15 on dist-test-slave-c7tk"
server_key: "06fa411d0dfd2be3a3eff7f4b1a03433"
server_key_iv: "28c1e6e10c19ba26f008c885f5e6192d"
server_key_version: "encryptionkey@0"
I20260924 21:08:15.357190 3402 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790284076800039-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790284076800039-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790284076800039-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 21:08:15.366078 3402 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 21:08:15.367617 3402 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 21:08:15.367776 3402 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 21:08:15.371042 3402 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45889
I20260924 21:08:15.371068 3469 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45889 every 8 connection(s)
I20260924 21:08:15.371330 3402 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790284076800039-392-0/minicluster-data/master-0/data/info.pb
I20260924 21:08:15.372121 3470 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 21:08:15.374099 3470 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 679f0a58106744c4a6db809ddfe48384: Bootstrap starting.
I20260924 21:08:15.374603 3470 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 679f0a58106744c4a6db809ddfe48384: Neither blocks nor log segments found. Creating new log.
I20260924 21:08:15.374873 3470 log.cc:824] T 00000000000000000000000000000000 P 679f0a58106744c4a6db809ddfe48384: Log is configured to *not* fsync() on all Append() calls
I20260924 21:08:15.375416 3470 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 679f0a58106744c4a6db809ddfe48384: No bootstrap required, opened a new log
I20260924 21:08:15.376852 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAzNNjS/build/release/bin/kudu as pid 3402
I20260924 21:08:15.376786 3470 raft_consensus.cc:348] T 00000000000000000000000000000000 P 679f0a58106744c4a6db809ddfe48384 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "679f0a58106744c4a6db809ddfe48384" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45889 } }
I20260924 21:08:15.376909 3470 raft_consensus.cc:374] T 00000000000000000000000000000000 P 679f0a58106744c4a6db809ddfe48384 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 21:08:15.376931 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790284076800039-392-0/minicluster-data/master-0/wal/instance
I20260924 21:08:15.376942 3470 raft_consensus.cc:729] T 00000000000000000000000000000000 P 679f0a58106744c4a6db809ddfe48384 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 679f0a58106744c4a6db809ddfe48384, State: Initialized, Role: FOLLOWER
I20260924 21:08:15.377084 392 external_mini_cluster.cc:1428] Setting key 2cd06b3727d701c989c5ddde9b8a1e19
I20260924 21:08:15.377068 3470 consensus_queue.cc:260] T 00000000000000000000000000000000 P 679f0a58106744c4a6db809ddfe48384 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "679f0a58106744c4a6db809ddfe48384" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45889 } }
I20260924 21:08:15.377137 3470 raft_consensus.cc:388] T 00000000000000000000000000000000 P 679f0a58106744c4a6db809ddfe48384 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 21:08:15.377173 3470 raft_consensus.cc:482] T 00000000000000000000000000000000 P 679f0a58106744c4a6db809ddfe48384 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 21:08:15.377216 3470 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 679f0a58106744c4a6db809ddfe48384 [term 0 FOLLOWER]: Advancing to term 1
I20260924 21:08:15.377898 3470 raft_consensus.cc:504] T 00000000000000000000000000000000 P 679f0a58106744c4a6db809ddfe48384 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "679f0a58106744c4a6db809ddfe48384" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45889 } }
I20260924 21:08:15.378002 3470 leader_election.cc:302] T 00000000000000000000000000000000 P 679f0a58106744c4a6db809ddfe48384 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 679f0a58106744c4a6db809ddfe48384; no voters:
I20260924 21:08:15.378146 3470 leader_election.cc:288] T 00000000000000000000000000000000 P 679f0a58106744c4a6db809ddfe48384 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 21:08:15.378170 3475 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 679f0a58106744c4a6db809ddfe48384 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 21:08:15.378263 3475 raft_consensus.cc:686] T 00000000000000000000000000000000 P 679f0a58106744c4a6db809ddfe48384 [term 1 LEADER]: Becoming Leader. State: Replica: 679f0a58106744c4a6db809ddfe48384, State: Running, Role: LEADER
I20260924 21:08:15.378379 3470 sys_catalog.cc:564] T 00000000000000000000000000000000 P 679f0a58106744c4a6db809ddfe48384 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 21:08:15.378388 3475 consensus_queue.cc:237] T 00000000000000000000000000000000 P 679f0a58106744c4a6db809ddfe48384 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "679f0a58106744c4a6db809ddfe48384" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45889 } }
I20260924 21:08:15.378831 3476 sys_catalog.cc:455] T 00000000000000000000000000000000 P 679f0a58106744c4a6db809ddfe48384 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "679f0a58106744c4a6db809ddfe48384" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "679f0a58106744c4a6db809ddfe48384" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45889 } } }
I20260924 21:08:15.378863 3477 sys_catalog.cc:455] T 00000000000000000000000000000000 P 679f0a58106744c4a6db809ddfe48384 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 679f0a58106744c4a6db809ddfe48384. Latest consensus state: current_term: 1 leader_uuid: "679f0a58106744c4a6db809ddfe48384" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "679f0a58106744c4a6db809ddfe48384" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45889 } } }
I20260924 21:08:15.379065 3477 sys_catalog.cc:458] T 00000000000000000000000000000000 P 679f0a58106744c4a6db809ddfe48384 [sys.catalog]: This master's current role is: LEADER
I20260924 21:08:15.379259 3484 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 21:08:15.378963 3476 sys_catalog.cc:458] T 00000000000000000000000000000000 P 679f0a58106744c4a6db809ddfe48384 [sys.catalog]: This master's current role is: LEADER
I20260924 21:08:15.379729 3484 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 21:08:15.382131 3484 catalog_manager.cc:1347] Generated new cluster ID: 22fd84c0b1d049ec8f466458976e4c82
I20260924 21:08:15.382176 3484 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 24 21:08:15 dist-test-slave-c7tk krb5kdc[3389](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284095, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 21:08:15.388135 3474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:15.377477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57808 (local address 127.0.98.62:45889)
0924 21:08:15.378021 (+ 544us) server_negotiation.cc:206] Beginning negotiation
0924 21:08:15.378030 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 21:08:15.378059 (+ 29us) server_negotiation.cc:407] Connection header received
0924 21:08:15.378159 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:08:15.378163 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:08:15.378226 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:08:15.379281 (+ 1055us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:08:15.379368 (+ 87us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:15.380526 (+ 1158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:15.381485 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:15.381923 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:15.382014 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:15.384308 (+ 2294us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 21:08:15.384311 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 21:08:15.384311 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 21:08:15.384320 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 21:08:15.385965 (+ 1645us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:08:15.386291 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:08:15.386293 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:08:15.386294 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:08:15.386388 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:08:15.386581 (+ 193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:08:15.386582 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:08:15.386582 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:08:15.386817 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 21:08:15.387288 (+ 471us) server_negotiation.cc:1035] Waiting for connection context
0924 21:08:15.387815 (+ 527us) server_negotiation.cc:299] Negotiation successful
0924 21:08:15.387944 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":497,"thread_start_us":449,"threads_started":1}
I20260924 21:08:15.393620 3484 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 21:08:15.394101 3484 catalog_manager.cc:1504] Loading token signing keys...
I20260924 21:08:15.400120 3484 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 679f0a58106744c4a6db809ddfe48384: Generated new TSK 0
I20260924 21:08:15.400251 3484 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790284076800039-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790284076800039-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260924 21:08:15.426203 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAzNNjS/build/release/bin/kudu
/tmp/dist-test-taskAzNNjS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790284076800039-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790284076800039-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790284076800039-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790284076800039-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45889
--keytab_file=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790284076800039-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33643
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790284076800039-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790284076800039-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790284076800039-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 21:08:15.502758 3496 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 21:08:15.502920 3496 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 21:08:15.502939 3496 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 21:08:15.503949 3496 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 21:08:15.503983 3496 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 21:08:15.504005 3496 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 21:08:15.505172 3496 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33643
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790284076800039-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790284076800039-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790284076800039-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790284076800039-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45889
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3496
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790284076800039-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 19:49:06 UTC on ccycloud.quasar-vhqjoc.root.comops.site
build id 4670864
I20260924 21:08:15.505357 3496 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 21:08:15.505559 3496 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 21:08:15.510051 3496 server_base.cc:1034] running on GCE node
W20260924 21:08:15.510073 3504 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 21:08:15.510118 3502 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 21:08:15.510160 3501 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 21:08:15.510414 3496 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 21:08:15.510566 3496 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 21:08:15.511682 3496 hybrid_clock.cc:648] HybridClock initialized: now 1790284095511671 us; error 29 us; skew 500 ppm
Sep 24 21:08:15 dist-test-slave-c7tk krb5kdc[3389](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284095, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 21:08:15.513782 3496 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 21:08:15.514200 3496 webserver.cc:460] Webserver started at http://127.0.98.1:38445/ using document root <none> and password file <none>
I20260924 21:08:15.514385 3496 fs_manager.cc:359] Metadata directory not provided
I20260924 21:08:15.514447 3496 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 21:08:15.514564 3496 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 21:08:15.515440 3496 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790284076800039-392-0/minicluster-data/ts-0/data/instance:
uuid: "415ff88c90974ed0954000ff5a7c47c0"
format_stamp: "Formatted at 2026-09-24 21:08:15 on dist-test-slave-c7tk"
server_key: "c89375861722f9d93f58fe3adada22a7"
server_key_iv: "a1a6dab22d5fe9d33e1849438c2541c1"
server_key_version: "encryptionkey@0"
I20260924 21:08:15.515734 3496 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790284076800039-392-0/minicluster-data/ts-0/wal/instance:
uuid: "415ff88c90974ed0954000ff5a7c47c0"
format_stamp: "Formatted at 2026-09-24 21:08:15 on dist-test-slave-c7tk"
server_key: "c89375861722f9d93f58fe3adada22a7"
server_key_iv: "a1a6dab22d5fe9d33e1849438c2541c1"
server_key_version: "encryptionkey@0"
I20260924 21:08:15.516784 3496 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260924 21:08:15.517328 3511 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 21:08:15.517480 3496 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 21:08:15.517560 3496 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790284076800039-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790284076800039-392-0/minicluster-data/ts-0/wal
uuid: "415ff88c90974ed0954000ff5a7c47c0"
format_stamp: "Formatted at 2026-09-24 21:08:15 on dist-test-slave-c7tk"
server_key: "c89375861722f9d93f58fe3adada22a7"
server_key_iv: "a1a6dab22d5fe9d33e1849438c2541c1"
server_key_version: "encryptionkey@0"
I20260924 21:08:15.517629 3496 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790284076800039-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790284076800039-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790284076800039-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 21:08:15.531708 3496 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 21:08:15.533144 3496 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 21:08:15.533272 3496 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 21:08:15.533449 3496 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 21:08:15.533738 3496 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 21:08:15.533788 3496 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 21:08:15.533850 3496 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 21:08:15.533881 3496 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 21:08:15.539288 3496 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:34251
I20260924 21:08:15.539319 3624 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:34251 every 8 connection(s)
I20260924 21:08:15.539600 3496 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790284076800039-392-0/minicluster-data/ts-0/data/info.pb
I20260924 21:08:15.540172 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAzNNjS/build/release/bin/kudu as pid 3496
I20260924 21:08:15.540262 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790284076800039-392-0/minicluster-data/ts-0/wal/instance
I20260924 21:08:15.540483 392 external_mini_cluster.cc:1428] Setting key e2b95fac3d08d3f31572d410f0f0088d
Sep 24 21:08:15 dist-test-slave-c7tk krb5kdc[3389](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284095, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260924 21:08:15.548710 3474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:15.540236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:52361 (local address 127.0.98.62:45889)
0924 21:08:15.540278 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0924 21:08:15.540280 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 21:08:15.540575 (+ 295us) server_negotiation.cc:407] Connection header received
0924 21:08:15.541147 (+ 572us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:08:15.541148 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:08:15.541164 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:08:15.541212 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:08:15.542379 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:15.543095 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:15.543936 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:15.544240 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:15.544317 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:15.546134 (+ 1817us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 21:08:15.546136 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 21:08:15.546136 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 21:08:15.546143 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 21:08:15.547534 (+ 1391us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:08:15.547898 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:08:15.547900 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:08:15.547900 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:08:15.547957 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:08:15.548164 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:08:15.548165 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:08:15.548166 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:08:15.548305 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 21:08:15.548376 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0924 21:08:15.548594 (+ 218us) server_negotiation.cc:299] Negotiation successful
0924 21:08:15.548651 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260924 21:08:15.548835 3627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:15.540236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45889 (local address 127.0.98.1:52361)
0924 21:08:15.540390 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0924 21:08:15.540456 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0924 21:08:15.541031 (+ 575us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 21:08:15.541278 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 21:08:15.541282 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 21:08:15.542053 (+ 771us) client_negotiation.cc:189] Negotiated authn=SASL
0924 21:08:15.542286 (+ 233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:15.542291 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:15.543185 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:15.543185 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:15.543864 (+ 679us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:15.543865 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:15.544319 (+ 454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:15.544319 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:15.544464 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:15.544896 (+ 432us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 21:08:15.544903 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0924 21:08:15.546055 (+ 1152us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 21:08:15.547606 (+ 1551us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 21:08:15.547611 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 21:08:15.547612 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 21:08:15.547825 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 21:08:15.548021 (+ 196us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 21:08:15.548021 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 21:08:15.548022 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 21:08:15.548105 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 21:08:15.548368 (+ 263us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 21:08:15.548373 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 21:08:15.548473 (+ 100us) client_negotiation.cc:769] Sending connection context
0924 21:08:15.548553 (+ 80us) client_negotiation.cc:240] Negotiation successful
0924 21:08:15.548656 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":81,"threads_started":1}
I20260924 21:08:15.549263 3625 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45889
I20260924 21:08:15.549329 3625 heartbeater.cc:464] Registering TS with master...
I20260924 21:08:15.549444 3625 heartbeater.cc:511] Master 127.0.98.62:45889 requested a full tablet report, sending...
I20260924 21:08:15.549901 3434 ts_manager.cc:194] Registered new tserver with Master: 415ff88c90974ed0954000ff5a7c47c0 (127.0.98.1:34251)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260924 21:08:15.550511 3434 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:52361
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790284076800039-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790284076800039-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260924 21:08:15.562242 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAzNNjS/build/release/bin/kudu
/tmp/dist-test-taskAzNNjS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790284076800039-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790284076800039-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790284076800039-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790284076800039-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45889
--keytab_file=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790284076800039-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33643
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790284076800039-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790284076800039-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790284076800039-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 21:08:15.633338 3630 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 21:08:15.633494 3630 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 21:08:15.633513 3630 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 21:08:15.634621 3630 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 21:08:15.634661 3630 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 21:08:15.634712 3630 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 21:08:15.635917 3630 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33643
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790284076800039-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790284076800039-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790284076800039-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790284076800039-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45889
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3630
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790284076800039-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 19:49:06 UTC on ccycloud.quasar-vhqjoc.root.comops.site
build id 4670864
I20260924 21:08:15.636112 3630 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 21:08:15.636293 3630 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 21:08:15.640799 3638 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 21:08:15.640836 3635 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 21:08:15.640957 3630 server_base.cc:1034] running on GCE node
W20260924 21:08:15.640808 3636 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 21:08:15.641173 3630 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 21:08:15.641389 3630 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 21:08:15.642513 3630 hybrid_clock.cc:648] HybridClock initialized: now 1790284095642484 us; error 46 us; skew 500 ppm
Sep 24 21:08:15 dist-test-slave-c7tk krb5kdc[3389](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284095, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 21:08:15.644821 3630 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 21:08:15.645187 3630 webserver.cc:460] Webserver started at http://127.0.98.2:37851/ using document root <none> and password file <none>
I20260924 21:08:15.645375 3630 fs_manager.cc:359] Metadata directory not provided
I20260924 21:08:15.645434 3630 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 21:08:15.645573 3630 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 21:08:15.646473 3630 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790284076800039-392-0/minicluster-data/ts-1/data/instance:
uuid: "e1aa888252c146d3ba47bc7a31fd6aef"
format_stamp: "Formatted at 2026-09-24 21:08:15 on dist-test-slave-c7tk"
server_key: "53b4cba0a8499128777034417a8fbc39"
server_key_iv: "32eaaa4af34a261f3620d25721ee1273"
server_key_version: "encryptionkey@0"
I20260924 21:08:15.646786 3630 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790284076800039-392-0/minicluster-data/ts-1/wal/instance:
uuid: "e1aa888252c146d3ba47bc7a31fd6aef"
format_stamp: "Formatted at 2026-09-24 21:08:15 on dist-test-slave-c7tk"
server_key: "53b4cba0a8499128777034417a8fbc39"
server_key_iv: "32eaaa4af34a261f3620d25721ee1273"
server_key_version: "encryptionkey@0"
I20260924 21:08:15.647931 3630 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260924 21:08:15.648569 3645 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 21:08:15.648727 3630 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 21:08:15.648826 3630 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790284076800039-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790284076800039-392-0/minicluster-data/ts-1/wal
uuid: "e1aa888252c146d3ba47bc7a31fd6aef"
format_stamp: "Formatted at 2026-09-24 21:08:15 on dist-test-slave-c7tk"
server_key: "53b4cba0a8499128777034417a8fbc39"
server_key_iv: "32eaaa4af34a261f3620d25721ee1273"
server_key_version: "encryptionkey@0"
I20260924 21:08:15.648918 3630 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790284076800039-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790284076800039-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790284076800039-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 21:08:15.666433 3630 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 21:08:15.667987 3630 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 21:08:15.668138 3630 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 21:08:15.668347 3630 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 21:08:15.668625 3630 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 21:08:15.668674 3630 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 21:08:15.668708 3630 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 21:08:15.668735 3630 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 21:08:15.673674 3630 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:36597
I20260924 21:08:15.673708 3758 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:36597 every 8 connection(s)
I20260924 21:08:15.674023 3630 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790284076800039-392-0/minicluster-data/ts-1/data/info.pb
I20260924 21:08:15.675899 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAzNNjS/build/release/bin/kudu as pid 3630
I20260924 21:08:15.675995 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790284076800039-392-0/minicluster-data/ts-1/wal/instance
I20260924 21:08:15.676182 392 external_mini_cluster.cc:1428] Setting key 799ee18a8263bb025d5a1e6b50a59613
Sep 24 21:08:15 dist-test-slave-c7tk krb5kdc[3389](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284095, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 21:08:15.682860 3474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:15.674658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:53617 (local address 127.0.98.62:45889)
0924 21:08:15.674700 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0924 21:08:15.674702 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 21:08:15.674888 (+ 186us) server_negotiation.cc:407] Connection header received
0924 21:08:15.675302 (+ 414us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:08:15.675303 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:08:15.675310 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:08:15.675355 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:08:15.676145 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:15.676733 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:15.677596 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:15.677923 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:15.677990 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:15.680284 (+ 2294us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 21:08:15.680286 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 21:08:15.680286 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 21:08:15.680293 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 21:08:15.681718 (+ 1425us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:08:15.682077 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:08:15.682078 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:08:15.682079 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:08:15.682138 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:08:15.682326 (+ 188us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:08:15.682327 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:08:15.682327 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:08:15.682448 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 21:08:15.682504 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0924 21:08:15.682701 (+ 197us) server_negotiation.cc:299] Negotiation successful
0924 21:08:15.682772 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 21:08:15.682879 3761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:15.674625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45889 (local address 127.0.98.2:53617)
0924 21:08:15.674765 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0924 21:08:15.674814 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0924 21:08:15.675221 (+ 407us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 21:08:15.675403 (+ 182us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 21:08:15.675410 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 21:08:15.675866 (+ 456us) client_negotiation.cc:189] Negotiated authn=SASL
0924 21:08:15.676059 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:15.676064 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:15.676824 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:15.676825 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:15.677519 (+ 694us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:15.677521 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:15.678003 (+ 482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:15.678004 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:15.678155 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:15.678650 (+ 495us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 21:08:15.678659 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 21:08:15.680201 (+ 1542us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 21:08:15.681787 (+ 1586us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 21:08:15.681792 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 21:08:15.681826 (+ 34us) client_negotiation.cc:760] calling sasl_client_step()
0924 21:08:15.682012 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 21:08:15.682201 (+ 189us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 21:08:15.682201 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 21:08:15.682202 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 21:08:15.682278 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 21:08:15.682507 (+ 229us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 21:08:15.682510 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 21:08:15.682599 (+ 89us) client_negotiation.cc:769] Sending connection context
0924 21:08:15.682662 (+ 63us) client_negotiation.cc:240] Negotiation successful
0924 21:08:15.682756 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":75,"threads_started":1}
I20260924 21:08:15.683281 3759 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45889
I20260924 21:08:15.683370 3759 heartbeater.cc:464] Registering TS with master...
I20260924 21:08:15.683530 3759 heartbeater.cc:511] Master 127.0.98.62:45889 requested a full tablet report, sending...
I20260924 21:08:15.683848 3434 ts_manager.cc:194] Registered new tserver with Master: e1aa888252c146d3ba47bc7a31fd6aef (127.0.98.2:36597)
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260924 21:08:15.684204 3434 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:53617
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790284076800039-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790284076800039-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260924 21:08:15.697933 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAzNNjS/build/release/bin/kudu
/tmp/dist-test-taskAzNNjS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790284076800039-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790284076800039-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790284076800039-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790284076800039-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45889
--keytab_file=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790284076800039-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33643
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790284076800039-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790284076800039-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790284076800039-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 21:08:15.767598 3764 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 21:08:15.767788 3764 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 21:08:15.767810 3764 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 21:08:15.768953 3764 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 21:08:15.768991 3764 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 21:08:15.769038 3764 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 21:08:15.770285 3764 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33643
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790284076800039-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790284076800039-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790284076800039-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790284076800039-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45889
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3764
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790284076800039-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 19:49:06 UTC on ccycloud.quasar-vhqjoc.root.comops.site
build id 4670864
I20260924 21:08:15.770498 3764 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 21:08:15.770692 3764 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 21:08:15.775229 3772 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 21:08:15.775198 3769 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 21:08:15.775174 3770 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 21:08:15.775168 3764 server_base.cc:1034] running on GCE node
I20260924 21:08:15.775585 3764 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 21:08:15.775789 3764 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 21:08:15.776932 3764 hybrid_clock.cc:648] HybridClock initialized: now 1790284095776923 us; error 25 us; skew 500 ppm
Sep 24 21:08:15 dist-test-slave-c7tk krb5kdc[3389](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284095, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 21:08:15.779467 3764 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 21:08:15.779838 3764 webserver.cc:460] Webserver started at http://127.0.98.3:35263/ using document root <none> and password file <none>
I20260924 21:08:15.780038 3764 fs_manager.cc:359] Metadata directory not provided
I20260924 21:08:15.780092 3764 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 21:08:15.780195 3764 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 21:08:15.781042 3764 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790284076800039-392-0/minicluster-data/ts-2/data/instance:
uuid: "f23258da3dea4db0970ed83c07102e73"
format_stamp: "Formatted at 2026-09-24 21:08:15 on dist-test-slave-c7tk"
server_key: "f1e4b2342f2eae1b044fbccce7157efc"
server_key_iv: "870c15112f5262dff029aeb04733bb4f"
server_key_version: "encryptionkey@0"
I20260924 21:08:15.781371 3764 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790284076800039-392-0/minicluster-data/ts-2/wal/instance:
uuid: "f23258da3dea4db0970ed83c07102e73"
format_stamp: "Formatted at 2026-09-24 21:08:15 on dist-test-slave-c7tk"
server_key: "f1e4b2342f2eae1b044fbccce7157efc"
server_key_iv: "870c15112f5262dff029aeb04733bb4f"
server_key_version: "encryptionkey@0"
I20260924 21:08:15.782521 3764 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20260924 21:08:15.783099 3779 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 21:08:15.783349 3764 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 21:08:15.783425 3764 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790284076800039-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790284076800039-392-0/minicluster-data/ts-2/wal
uuid: "f23258da3dea4db0970ed83c07102e73"
format_stamp: "Formatted at 2026-09-24 21:08:15 on dist-test-slave-c7tk"
server_key: "f1e4b2342f2eae1b044fbccce7157efc"
server_key_iv: "870c15112f5262dff029aeb04733bb4f"
server_key_version: "encryptionkey@0"
I20260924 21:08:15.783509 3764 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790284076800039-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790284076800039-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790284076800039-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 21:08:15.801688 3764 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 21:08:15.803524 3764 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 21:08:15.803655 3764 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 21:08:15.803834 3764 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 21:08:15.804103 3764 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 21:08:15.804144 3764 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 21:08:15.804173 3764 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 21:08:15.804195 3764 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 21:08:15.809298 3764 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:44007
I20260924 21:08:15.809358 3892 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:44007 every 8 connection(s)
I20260924 21:08:15.809616 3764 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790284076800039-392-0/minicluster-data/ts-2/data/info.pb
I20260924 21:08:15.811496 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAzNNjS/build/release/bin/kudu as pid 3764
I20260924 21:08:15.811566 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790284076800039-392-0/minicluster-data/ts-2/wal/instance
I20260924 21:08:15.811733 392 external_mini_cluster.cc:1428] Setting key dbce981e050484312e6596e6cd3f54d6
Sep 24 21:08:15 dist-test-slave-c7tk krb5kdc[3389](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284095, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 21:08:15.817770 3474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:15.810216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:50609 (local address 127.0.98.62:45889)
0924 21:08:15.810253 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0924 21:08:15.810255 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 21:08:15.810429 (+ 174us) server_negotiation.cc:407] Connection header received
0924 21:08:15.810908 (+ 479us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:08:15.810909 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:08:15.810921 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:08:15.810969 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:08:15.811843 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:15.812451 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:15.813380 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:15.813599 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:15.813673 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:15.815569 (+ 1896us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 21:08:15.815571 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 21:08:15.815571 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 21:08:15.815577 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 21:08:15.816598 (+ 1021us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:08:15.816932 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:08:15.816934 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:08:15.816934 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:08:15.816985 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:08:15.817198 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:08:15.817199 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:08:15.817200 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:08:15.817301 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 21:08:15.817367 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0924 21:08:15.817595 (+ 228us) server_negotiation.cc:299] Negotiation successful
0924 21:08:15.817665 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260924 21:08:15.817979 3895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:15.810193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45889 (local address 127.0.98.3:50609)
0924 21:08:15.810336 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0924 21:08:15.810364 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0924 21:08:15.810817 (+ 453us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 21:08:15.811036 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 21:08:15.811039 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 21:08:15.811503 (+ 464us) client_negotiation.cc:189] Negotiated authn=SASL
0924 21:08:15.811728 (+ 225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:15.811734 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:15.812528 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:15.812529 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:15.813280 (+ 751us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:15.813282 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:15.813678 (+ 396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:15.813680 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:15.813834 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:15.814301 (+ 467us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 21:08:15.814310 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 21:08:15.815481 (+ 1171us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 21:08:15.816676 (+ 1195us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 21:08:15.816682 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 21:08:15.816683 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 21:08:15.816868 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 21:08:15.817047 (+ 179us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 21:08:15.817047 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 21:08:15.817048 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 21:08:15.817142 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 21:08:15.817358 (+ 216us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 21:08:15.817363 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 21:08:15.817484 (+ 121us) client_negotiation.cc:769] Sending connection context
0924 21:08:15.817576 (+ 92us) client_negotiation.cc:240] Negotiation successful
0924 21:08:15.817730 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":75,"threads_started":1}
I20260924 21:08:15.818326 3893 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45889
I20260924 21:08:15.818425 3893 heartbeater.cc:464] Registering TS with master...
I20260924 21:08:15.818583 3893 heartbeater.cc:511] Master 127.0.98.62:45889 requested a full tablet report, sending...
I20260924 21:08:15.818917 3434 ts_manager.cc:194] Registered new tserver with Master: f23258da3dea4db0970ed83c07102e73 (127.0.98.3:44007)
I20260924 21:08:15.819288 3434 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:50609
I20260924 21:08:15.823153 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 24 21:08:15 dist-test-slave-c7tk krb5kdc[3389](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284095, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 24 21:08:15 dist-test-slave-c7tk krb5kdc[3389](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284095, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 21:08:15.842312 3474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:15.836568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57814 (local address 127.0.98.62:45889)
0924 21:08:15.836618 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0924 21:08:15.836620 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 21:08:15.836748 (+ 128us) server_negotiation.cc:407] Connection header received
0924 21:08:15.836785 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:08:15.836785 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:08:15.836792 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:08:15.836834 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:08:15.837262 (+ 428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:15.837650 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:15.838378 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:15.838553 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:15.838611 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:15.840484 (+ 1873us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 21:08:15.840485 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 21:08:15.840486 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 21:08:15.840491 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 21:08:15.841480 (+ 989us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:08:15.841728 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:08:15.841728 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:08:15.841729 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:08:15.841769 (+ 40us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:08:15.841956 (+ 187us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:08:15.841956 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:08:15.841956 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:08:15.842061 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 21:08:15.842121 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0924 21:08:15.842193 (+ 72us) server_negotiation.cc:299] Negotiation successful
0924 21:08:15.842240 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
W20260924 21:08:15.842516 3434 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:57814
I20260924 21:08:15.842873 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAzNNjS/build/release/bin/kudu with pid 3496
I20260924 21:08:15.846697 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAzNNjS/build/release/bin/kudu with pid 3630
I20260924 21:08:15.851224 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAzNNjS/build/release/bin/kudu with pid 3764
I20260924 21:08:15.855012 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAzNNjS/build/release/bin/kudu with pid 3402
2026-09-24T21:08:15Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (1610 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790284076800039-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 21:08:15 dist-test-slave-c7tk krb5kdc[3905](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 21:08:15 dist-test-slave-c7tk krb5kdc[3905](info): set up 2 sockets
Sep 24 21:08:15 dist-test-slave-c7tk krb5kdc[3905](info): commencing operation
krb5kdc: starting...
W20260924 21:08:16.781280 392 mini_kdc.cc:121] Time spent starting KDC: real 0.907s user 0.002s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790284076800039-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790284076800039-392-0/krb5kdc/test-user.keytab.
Sep 24 21:08:16 dist-test-slave-c7tk krb5kdc[3905](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284096, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T21:08:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T21:08:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790284076800039-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790284076800039-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 21:08:16.882411 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAzNNjS/build/release/bin/kudu
/tmp/dist-test-taskAzNNjS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790284076800039-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790284076800039-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790284076800039-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790284076800039-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41447
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790284076800039-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35941
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41447
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790284076800039-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790284076800039-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790284076800039-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 21:08:16.952915 3918 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 21:08:16.953101 3918 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 21:08:16.953135 3918 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 21:08:16.954195 3918 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 21:08:16.954244 3918 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 21:08:16.954264 3918 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 21:08:16.954283 3918 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 21:08:16.954319 3918 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 21:08:16.955623 3918 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35941
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790284076800039-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790284076800039-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41447
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790284076800039-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41447
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790284076800039-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3918
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790284076800039-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 19:49:06 UTC on ccycloud.quasar-vhqjoc.root.comops.site
build id 4670864
I20260924 21:08:16.955842 3918 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 21:08:16.956038 3918 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 21:08:16.960359 3926 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 21:08:16.960391 3923 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 21:08:16.960369 3924 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 21:08:16.960502 3918 server_base.cc:1034] running on GCE node
I20260924 21:08:16.960840 3918 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 21:08:16.961059 3918 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 21:08:16.962198 3918 hybrid_clock.cc:648] HybridClock initialized: now 1790284096962191 us; error 30 us; skew 500 ppm
Sep 24 21:08:16 dist-test-slave-c7tk krb5kdc[3905](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284096, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 21:08:16.964169 3918 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 21:08:16.964497 3918 webserver.cc:460] Webserver started at http://127.0.98.62:45025/ using document root <none> and password file <none>
I20260924 21:08:16.964681 3918 fs_manager.cc:359] Metadata directory not provided
I20260924 21:08:16.964735 3918 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 21:08:16.964851 3918 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 21:08:16.965691 3918 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790284076800039-392-0/minicluster-data/master-0/data/instance:
uuid: "58e43ba8341b49adaf4f7b30c2339f03"
format_stamp: "Formatted at 2026-09-24 21:08:16 on dist-test-slave-c7tk"
server_key: "d628fa9442d4ae4b154841faa269cef6"
server_key_iv: "8777e7f586634c0187c6ac093bb4bf79"
server_key_version: "encryptionkey@0"
I20260924 21:08:16.966081 3918 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790284076800039-392-0/minicluster-data/master-0/wal/instance:
uuid: "58e43ba8341b49adaf4f7b30c2339f03"
format_stamp: "Formatted at 2026-09-24 21:08:16 on dist-test-slave-c7tk"
server_key: "d628fa9442d4ae4b154841faa269cef6"
server_key_iv: "8777e7f586634c0187c6ac093bb4bf79"
server_key_version: "encryptionkey@0"
I20260924 21:08:16.967196 3918 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 21:08:16.967773 3933 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 21:08:16.967965 3918 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 21:08:16.968053 3918 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790284076800039-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790284076800039-392-0/minicluster-data/master-0/wal
uuid: "58e43ba8341b49adaf4f7b30c2339f03"
format_stamp: "Formatted at 2026-09-24 21:08:16 on dist-test-slave-c7tk"
server_key: "d628fa9442d4ae4b154841faa269cef6"
server_key_iv: "8777e7f586634c0187c6ac093bb4bf79"
server_key_version: "encryptionkey@0"
I20260924 21:08:16.968129 3918 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790284076800039-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790284076800039-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790284076800039-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 21:08:16.976864 3918 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 21:08:16.978498 3918 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 21:08:16.978637 3918 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 21:08:16.982020 3918 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41447
I20260924 21:08:16.982060 3985 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41447 every 8 connection(s)
I20260924 21:08:16.982344 3918 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790284076800039-392-0/minicluster-data/master-0/data/info.pb
I20260924 21:08:16.983040 3986 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 21:08:16.984804 3986 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 58e43ba8341b49adaf4f7b30c2339f03: Bootstrap starting.
I20260924 21:08:16.985266 3986 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 58e43ba8341b49adaf4f7b30c2339f03: Neither blocks nor log segments found. Creating new log.
I20260924 21:08:16.985507 3986 log.cc:824] T 00000000000000000000000000000000 P 58e43ba8341b49adaf4f7b30c2339f03: Log is configured to *not* fsync() on all Append() calls
I20260924 21:08:16.986007 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAzNNjS/build/release/bin/kudu as pid 3918
I20260924 21:08:16.986065 3986 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 58e43ba8341b49adaf4f7b30c2339f03: No bootstrap required, opened a new log
I20260924 21:08:16.986101 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790284076800039-392-0/minicluster-data/master-0/wal/instance
I20260924 21:08:16.986254 392 external_mini_cluster.cc:1428] Setting key fc02d0be68fe84613f626bd08843e4dc
I20260924 21:08:16.988049 3986 raft_consensus.cc:348] T 00000000000000000000000000000000 P 58e43ba8341b49adaf4f7b30c2339f03 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58e43ba8341b49adaf4f7b30c2339f03" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41447 } }
I20260924 21:08:16.988183 3986 raft_consensus.cc:374] T 00000000000000000000000000000000 P 58e43ba8341b49adaf4f7b30c2339f03 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 21:08:16.988229 3986 raft_consensus.cc:729] T 00000000000000000000000000000000 P 58e43ba8341b49adaf4f7b30c2339f03 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58e43ba8341b49adaf4f7b30c2339f03, State: Initialized, Role: FOLLOWER
I20260924 21:08:16.988353 3986 consensus_queue.cc:260] T 00000000000000000000000000000000 P 58e43ba8341b49adaf4f7b30c2339f03 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58e43ba8341b49adaf4f7b30c2339f03" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41447 } }
I20260924 21:08:16.988441 3986 raft_consensus.cc:388] T 00000000000000000000000000000000 P 58e43ba8341b49adaf4f7b30c2339f03 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 21:08:16.988484 3986 raft_consensus.cc:482] T 00000000000000000000000000000000 P 58e43ba8341b49adaf4f7b30c2339f03 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 21:08:16.988530 3986 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 58e43ba8341b49adaf4f7b30c2339f03 [term 0 FOLLOWER]: Advancing to term 1
I20260924 21:08:16.989015 3986 raft_consensus.cc:504] T 00000000000000000000000000000000 P 58e43ba8341b49adaf4f7b30c2339f03 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58e43ba8341b49adaf4f7b30c2339f03" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41447 } }
I20260924 21:08:16.989120 3986 leader_election.cc:302] T 00000000000000000000000000000000 P 58e43ba8341b49adaf4f7b30c2339f03 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 58e43ba8341b49adaf4f7b30c2339f03; no voters:
I20260924 21:08:16.989245 3986 leader_election.cc:288] T 00000000000000000000000000000000 P 58e43ba8341b49adaf4f7b30c2339f03 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 21:08:16.989274 3991 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 58e43ba8341b49adaf4f7b30c2339f03 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 21:08:16.989447 3991 raft_consensus.cc:686] T 00000000000000000000000000000000 P 58e43ba8341b49adaf4f7b30c2339f03 [term 1 LEADER]: Becoming Leader. State: Replica: 58e43ba8341b49adaf4f7b30c2339f03, State: Running, Role: LEADER
I20260924 21:08:16.989530 3991 consensus_queue.cc:237] T 00000000000000000000000000000000 P 58e43ba8341b49adaf4f7b30c2339f03 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58e43ba8341b49adaf4f7b30c2339f03" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41447 } }
I20260924 21:08:16.989439 3986 sys_catalog.cc:564] T 00000000000000000000000000000000 P 58e43ba8341b49adaf4f7b30c2339f03 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 21:08:16.989931 3993 sys_catalog.cc:455] T 00000000000000000000000000000000 P 58e43ba8341b49adaf4f7b30c2339f03 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "58e43ba8341b49adaf4f7b30c2339f03" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58e43ba8341b49adaf4f7b30c2339f03" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41447 } } }
I20260924 21:08:16.990020 3993 sys_catalog.cc:458] T 00000000000000000000000000000000 P 58e43ba8341b49adaf4f7b30c2339f03 [sys.catalog]: This master's current role is: LEADER
I20260924 21:08:16.989835 3992 sys_catalog.cc:455] T 00000000000000000000000000000000 P 58e43ba8341b49adaf4f7b30c2339f03 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 58e43ba8341b49adaf4f7b30c2339f03. Latest consensus state: current_term: 1 leader_uuid: "58e43ba8341b49adaf4f7b30c2339f03" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58e43ba8341b49adaf4f7b30c2339f03" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41447 } } }
I20260924 21:08:16.990135 3992 sys_catalog.cc:458] T 00000000000000000000000000000000 P 58e43ba8341b49adaf4f7b30c2339f03 [sys.catalog]: This master's current role is: LEADER
I20260924 21:08:16.990340 3995 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 21:08:16.990796 3995 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 24 21:08:16 dist-test-slave-c7tk krb5kdc[3905](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284096, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 21:08:16.994169 3995 catalog_manager.cc:1347] Generated new cluster ID: b15015cd4f584ac685a67b680b4596d3
I20260924 21:08:16.994218 3995 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 21:08:16.995954 3990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:16.986633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46596 (local address 127.0.98.62:41447)
0924 21:08:16.986769 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0924 21:08:16.986776 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 21:08:16.986804 (+ 28us) server_negotiation.cc:407] Connection header received
0924 21:08:16.987137 (+ 333us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:08:16.987142 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:08:16.987181 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:08:16.987271 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:08:16.987949 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:16.989154 (+ 1205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:16.990081 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:16.990691 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:16.990781 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:16.993111 (+ 2330us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 21:08:16.993114 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 21:08:16.993114 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 21:08:16.993124 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0924 21:08:16.994750 (+ 1626us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:08:16.995026 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:08:16.995028 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:08:16.995029 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:08:16.995109 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:08:16.995256 (+ 147us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:08:16.995257 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:08:16.995257 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:08:16.995490 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 21:08:16.995607 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0924 21:08:16.995655 (+ 48us) server_negotiation.cc:299] Negotiation successful
0924 21:08:16.995766 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99,"thread_start_us":77,"threads_started":1}
I20260924 21:08:17.000867 3995 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 21:08:17.001276 3995 catalog_manager.cc:1504] Loading token signing keys...
I20260924 21:08:17.004396 3995 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 58e43ba8341b49adaf4f7b30c2339f03: Generated new TSK 0
I20260924 21:08:17.004530 3995 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790284076800039-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790284076800039-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260924 21:08:17.028860 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAzNNjS/build/release/bin/kudu
/tmp/dist-test-taskAzNNjS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790284076800039-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790284076800039-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790284076800039-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790284076800039-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41447
--keytab_file=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790284076800039-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35941
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790284076800039-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790284076800039-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790284076800039-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 21:08:17.099659 4012 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 21:08:17.099826 4012 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 21:08:17.099846 4012 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 21:08:17.100920 4012 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 21:08:17.100957 4012 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 21:08:17.100980 4012 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 21:08:17.102274 4012 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35941
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790284076800039-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790284076800039-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790284076800039-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790284076800039-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41447
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4012
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790284076800039-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 19:49:06 UTC on ccycloud.quasar-vhqjoc.root.comops.site
build id 4670864
I20260924 21:08:17.102483 4012 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 21:08:17.102669 4012 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 21:08:17.107542 4018 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 21:08:17.107592 4017 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 21:08:17.107577 4020 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 21:08:17.107719 4012 server_base.cc:1034] running on GCE node
I20260924 21:08:17.107928 4012 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 21:08:17.108144 4012 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 21:08:17.109299 4012 hybrid_clock.cc:648] HybridClock initialized: now 1790284097109281 us; error 45 us; skew 500 ppm
Sep 24 21:08:17 dist-test-slave-c7tk krb5kdc[3905](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284097, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 21:08:17.111351 4012 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 21:08:17.111726 4012 webserver.cc:460] Webserver started at http://127.0.98.1:35557/ using document root <none> and password file <none>
I20260924 21:08:17.111910 4012 fs_manager.cc:359] Metadata directory not provided
I20260924 21:08:17.111977 4012 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 21:08:17.112059 4012 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 21:08:17.112850 4012 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790284076800039-392-0/minicluster-data/ts-0/data/instance:
uuid: "9b3c82239fb34dadaa664a14b3d2edd2"
format_stamp: "Formatted at 2026-09-24 21:08:17 on dist-test-slave-c7tk"
server_key: "f0dc67d4c7c185fcd3ff96158af89509"
server_key_iv: "be857153ba8fd90c75b692c1891eb1eb"
server_key_version: "encryptionkey@0"
I20260924 21:08:17.113113 4012 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790284076800039-392-0/minicluster-data/ts-0/wal/instance:
uuid: "9b3c82239fb34dadaa664a14b3d2edd2"
format_stamp: "Formatted at 2026-09-24 21:08:17 on dist-test-slave-c7tk"
server_key: "f0dc67d4c7c185fcd3ff96158af89509"
server_key_iv: "be857153ba8fd90c75b692c1891eb1eb"
server_key_version: "encryptionkey@0"
I20260924 21:08:17.114220 4012 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 21:08:17.114849 4027 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 21:08:17.114991 4012 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 21:08:17.115074 4012 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790284076800039-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790284076800039-392-0/minicluster-data/ts-0/wal
uuid: "9b3c82239fb34dadaa664a14b3d2edd2"
format_stamp: "Formatted at 2026-09-24 21:08:17 on dist-test-slave-c7tk"
server_key: "f0dc67d4c7c185fcd3ff96158af89509"
server_key_iv: "be857153ba8fd90c75b692c1891eb1eb"
server_key_version: "encryptionkey@0"
I20260924 21:08:17.115140 4012 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790284076800039-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790284076800039-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790284076800039-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 21:08:17.135877 4012 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 21:08:17.137313 4012 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 21:08:17.137456 4012 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 21:08:17.137638 4012 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 21:08:17.137957 4012 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 21:08:17.138008 4012 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 21:08:17.138042 4012 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 21:08:17.138068 4012 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 21:08:17.143234 4012 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:33225
I20260924 21:08:17.143276 4140 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:33225 every 8 connection(s)
I20260924 21:08:17.143518 4012 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790284076800039-392-0/minicluster-data/ts-0/data/info.pb
Sep 24 21:08:17 dist-test-slave-c7tk krb5kdc[3905](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284097, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 21:08:17.150987 3990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:17.144100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:40271 (local address 127.0.98.62:41447)
0924 21:08:17.144147 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0924 21:08:17.144149 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 21:08:17.144319 (+ 170us) server_negotiation.cc:407] Connection header received
0924 21:08:17.145002 (+ 683us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:08:17.145003 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:08:17.145012 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:08:17.145056 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:08:17.145486 (+ 430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:17.145957 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:17.146734 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:17.146907 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:17.146948 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:17.148729 (+ 1781us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 21:08:17.148730 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 21:08:17.148730 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 21:08:17.148736 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 21:08:17.149861 (+ 1125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:08:17.150154 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:08:17.150155 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:08:17.150155 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:08:17.150198 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:08:17.150426 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:08:17.150427 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:08:17.150427 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:08:17.150536 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 21:08:17.150594 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0924 21:08:17.150814 (+ 220us) server_negotiation.cc:299] Negotiation successful
0924 21:08:17.150908 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260924 21:08:17.151091 4143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:17.144091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41447 (local address 127.0.98.1:40271)
0924 21:08:17.144217 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0924 21:08:17.144243 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0924 21:08:17.144931 (+ 688us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 21:08:17.145097 (+ 166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 21:08:17.145100 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 21:08:17.145282 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0924 21:08:17.145429 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:17.145432 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:17.146021 (+ 589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:17.146021 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:17.146669 (+ 648us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:17.146671 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:17.146960 (+ 289us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:17.146960 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:17.147040 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:17.147434 (+ 394us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 21:08:17.147440 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0924 21:08:17.148654 (+ 1214us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 21:08:17.149930 (+ 1276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 21:08:17.149934 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 21:08:17.149936 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 21:08:17.150102 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 21:08:17.150270 (+ 168us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 21:08:17.150271 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 21:08:17.150271 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 21:08:17.150361 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 21:08:17.150597 (+ 236us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 21:08:17.150603 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 21:08:17.150709 (+ 106us) client_negotiation.cc:769] Sending connection context
0924 21:08:17.150794 (+ 85us) client_negotiation.cc:240] Negotiation successful
0924 21:08:17.150917 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":79,"threads_started":1}
I20260924 21:08:17.151566 4141 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41447
I20260924 21:08:17.151644 4141 heartbeater.cc:464] Registering TS with master...
I20260924 21:08:17.151790 4141 heartbeater.cc:511] Master 127.0.98.62:41447 requested a full tablet report, sending...
I20260924 21:08:17.152202 3950 ts_manager.cc:194] Registered new tserver with Master: 9b3c82239fb34dadaa664a14b3d2edd2 (127.0.98.1:33225)
I20260924 21:08:17.152467 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAzNNjS/build/release/bin/kudu as pid 4012
I20260924 21:08:17.152559 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790284076800039-392-0/minicluster-data/ts-0/wal/instance
I20260924 21:08:17.152725 392 external_mini_cluster.cc:1428] Setting key daf64dfeedebafd6f9d5bc3fa0d2bf23
I20260924 21:08:17.152868 3950 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:40271
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790284076800039-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790284076800039-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260924 21:08:17.174278 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAzNNjS/build/release/bin/kudu
/tmp/dist-test-taskAzNNjS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790284076800039-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790284076800039-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790284076800039-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790284076800039-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41447
--keytab_file=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790284076800039-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35941
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790284076800039-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790284076800039-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790284076800039-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 21:08:17.243547 4146 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 21:08:17.243718 4146 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 21:08:17.243752 4146 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 21:08:17.244719 4146 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 21:08:17.244773 4146 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 21:08:17.244809 4146 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 21:08:17.246061 4146 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35941
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790284076800039-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790284076800039-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790284076800039-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790284076800039-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41447
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4146
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790284076800039-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 19:49:06 UTC on ccycloud.quasar-vhqjoc.root.comops.site
build id 4670864
I20260924 21:08:17.246272 4146 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 21:08:17.246454 4146 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 21:08:17.250679 4154 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 21:08:17.250788 4152 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 21:08:17.250741 4146 server_base.cc:1034] running on GCE node
W20260924 21:08:17.250870 4151 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 21:08:17.251088 4146 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 21:08:17.251295 4146 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 21:08:17.252420 4146 hybrid_clock.cc:648] HybridClock initialized: now 1790284097252410 us; error 24 us; skew 500 ppm
Sep 24 21:08:17 dist-test-slave-c7tk krb5kdc[3905](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284097, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 21:08:17.254531 4146 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 21:08:17.254880 4146 webserver.cc:460] Webserver started at http://127.0.98.2:44583/ using document root <none> and password file <none>
I20260924 21:08:17.255069 4146 fs_manager.cc:359] Metadata directory not provided
I20260924 21:08:17.255125 4146 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 21:08:17.255250 4146 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 21:08:17.256050 4146 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790284076800039-392-0/minicluster-data/ts-1/data/instance:
uuid: "b1e221a7e2b945508bc1bbbe69ed789c"
format_stamp: "Formatted at 2026-09-24 21:08:17 on dist-test-slave-c7tk"
server_key: "70d0fcbb3cee00ee54fc04e6ff077764"
server_key_iv: "ccd7fddebfab79e5627afc3668a35760"
server_key_version: "encryptionkey@0"
I20260924 21:08:17.256333 4146 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790284076800039-392-0/minicluster-data/ts-1/wal/instance:
uuid: "b1e221a7e2b945508bc1bbbe69ed789c"
format_stamp: "Formatted at 2026-09-24 21:08:17 on dist-test-slave-c7tk"
server_key: "70d0fcbb3cee00ee54fc04e6ff077764"
server_key_iv: "ccd7fddebfab79e5627afc3668a35760"
server_key_version: "encryptionkey@0"
I20260924 21:08:17.257463 4146 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260924 21:08:17.258138 4161 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 21:08:17.258289 4146 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 21:08:17.258389 4146 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790284076800039-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790284076800039-392-0/minicluster-data/ts-1/wal
uuid: "b1e221a7e2b945508bc1bbbe69ed789c"
format_stamp: "Formatted at 2026-09-24 21:08:17 on dist-test-slave-c7tk"
server_key: "70d0fcbb3cee00ee54fc04e6ff077764"
server_key_iv: "ccd7fddebfab79e5627afc3668a35760"
server_key_version: "encryptionkey@0"
I20260924 21:08:17.258455 4146 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790284076800039-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790284076800039-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790284076800039-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 21:08:17.267727 4146 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 21:08:17.269297 4146 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 21:08:17.269428 4146 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 21:08:17.269591 4146 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 21:08:17.269856 4146 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 21:08:17.269905 4146 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 21:08:17.269938 4146 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 21:08:17.269963 4146 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 21:08:17.275190 4146 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:42375
I20260924 21:08:17.275235 4274 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:42375 every 8 connection(s)
I20260924 21:08:17.275465 4146 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790284076800039-392-0/minicluster-data/ts-1/data/info.pb
I20260924 21:08:17.277968 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAzNNjS/build/release/bin/kudu as pid 4146
I20260924 21:08:17.278079 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790284076800039-392-0/minicluster-data/ts-1/wal/instance
I20260924 21:08:17.278247 392 external_mini_cluster.cc:1428] Setting key 5afad69116c42ac47ed62eccd52d5d4e
Sep 24 21:08:17 dist-test-slave-c7tk krb5kdc[3905](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284097, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 21:08:17.283828 3990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:17.275969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:39285 (local address 127.0.98.62:41447)
0924 21:08:17.276026 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0924 21:08:17.276027 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 21:08:17.276190 (+ 163us) server_negotiation.cc:407] Connection header received
0924 21:08:17.276930 (+ 740us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:08:17.276930 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:08:17.276939 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:08:17.276993 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:08:17.277445 (+ 452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:17.277961 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:17.279339 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:17.279501 (+ 162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:17.279547 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:17.281396 (+ 1849us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 21:08:17.281397 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 21:08:17.281398 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 21:08:17.281403 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 21:08:17.282658 (+ 1255us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:08:17.283071 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:08:17.283072 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:08:17.283072 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:08:17.283115 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:08:17.283325 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:08:17.283326 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:08:17.283326 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:08:17.283406 (+ 80us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 21:08:17.283447 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0924 21:08:17.283724 (+ 277us) server_negotiation.cc:299] Negotiation successful
0924 21:08:17.283776 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260924 21:08:17.284021 4277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:17.275963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41447 (local address 127.0.98.2:39285)
0924 21:08:17.276087 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0924 21:08:17.276118 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0924 21:08:17.276851 (+ 733us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 21:08:17.277036 (+ 185us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 21:08:17.277039 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 21:08:17.277237 (+ 198us) client_negotiation.cc:189] Negotiated authn=SASL
0924 21:08:17.277382 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:17.277384 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:17.278050 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:17.278051 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:17.279265 (+ 1214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:17.279268 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:17.279569 (+ 301us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:17.279570 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:17.279694 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:17.280079 (+ 385us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 21:08:17.280087 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 21:08:17.281316 (+ 1229us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 21:08:17.282726 (+ 1410us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 21:08:17.282732 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 21:08:17.282734 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 21:08:17.283005 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 21:08:17.283154 (+ 149us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 21:08:17.283154 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 21:08:17.283155 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 21:08:17.283272 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 21:08:17.283461 (+ 189us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 21:08:17.283466 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 21:08:17.283605 (+ 139us) client_negotiation.cc:769] Sending connection context
0924 21:08:17.283702 (+ 97us) client_negotiation.cc:240] Negotiation successful
0924 21:08:17.283835 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":90,"thread_start_us":69,"threads_started":1}
I20260924 21:08:17.284376 4275 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41447
I20260924 21:08:17.284459 4275 heartbeater.cc:464] Registering TS with master...
I20260924 21:08:17.284593 4275 heartbeater.cc:511] Master 127.0.98.62:41447 requested a full tablet report, sending...
I20260924 21:08:17.284924 3949 ts_manager.cc:194] Registered new tserver with Master: b1e221a7e2b945508bc1bbbe69ed789c (127.0.98.2:42375)
I20260924 21:08:17.285233 3949 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:39285
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790284076800039-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790284076800039-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260924 21:08:17.300307 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAzNNjS/build/release/bin/kudu
/tmp/dist-test-taskAzNNjS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790284076800039-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790284076800039-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790284076800039-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790284076800039-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41447
--keytab_file=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790284076800039-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35941
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790284076800039-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790284076800039-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790284076800039-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 21:08:17.372398 4280 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 21:08:17.372555 4280 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 21:08:17.372573 4280 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 21:08:17.373591 4280 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 21:08:17.373627 4280 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 21:08:17.373667 4280 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 21:08:17.374989 4280 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35941
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790284076800039-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790284076800039-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790284076800039-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790284076800039-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41447
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4280
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790284076800039-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 19:49:06 UTC on ccycloud.quasar-vhqjoc.root.comops.site
build id 4670864
I20260924 21:08:17.375185 4280 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 21:08:17.375361 4280 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 21:08:17.379570 4288 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 21:08:17.379616 4285 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 21:08:17.379570 4286 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 21:08:17.379848 4280 server_base.cc:1034] running on GCE node
I20260924 21:08:17.380029 4280 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 21:08:17.380203 4280 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 21:08:17.381335 4280 hybrid_clock.cc:648] HybridClock initialized: now 1790284097381314 us; error 34 us; skew 500 ppm
Sep 24 21:08:17 dist-test-slave-c7tk krb5kdc[3905](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284097, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 21:08:17.383585 4280 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 21:08:17.384003 4280 webserver.cc:460] Webserver started at http://127.0.98.3:42559/ using document root <none> and password file <none>
I20260924 21:08:17.384188 4280 fs_manager.cc:359] Metadata directory not provided
I20260924 21:08:17.384229 4280 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 21:08:17.384361 4280 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 21:08:17.385210 4280 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790284076800039-392-0/minicluster-data/ts-2/data/instance:
uuid: "483d31d21fce400881001c0b9cd80abc"
format_stamp: "Formatted at 2026-09-24 21:08:17 on dist-test-slave-c7tk"
server_key: "439253e0d92b331dddc69572ba3cd6e5"
server_key_iv: "d345f1e80ed74e42d7e467c359622907"
server_key_version: "encryptionkey@0"
I20260924 21:08:17.385527 4280 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790284076800039-392-0/minicluster-data/ts-2/wal/instance:
uuid: "483d31d21fce400881001c0b9cd80abc"
format_stamp: "Formatted at 2026-09-24 21:08:17 on dist-test-slave-c7tk"
server_key: "439253e0d92b331dddc69572ba3cd6e5"
server_key_iv: "d345f1e80ed74e42d7e467c359622907"
server_key_version: "encryptionkey@0"
I20260924 21:08:17.386794 4280 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 21:08:17.387470 4295 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 21:08:17.387624 4280 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 21:08:17.387703 4280 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790284076800039-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790284076800039-392-0/minicluster-data/ts-2/wal
uuid: "483d31d21fce400881001c0b9cd80abc"
format_stamp: "Formatted at 2026-09-24 21:08:17 on dist-test-slave-c7tk"
server_key: "439253e0d92b331dddc69572ba3cd6e5"
server_key_iv: "d345f1e80ed74e42d7e467c359622907"
server_key_version: "encryptionkey@0"
I20260924 21:08:17.387766 4280 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790284076800039-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790284076800039-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790284076800039-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 21:08:17.406515 4280 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 21:08:17.407949 4280 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 21:08:17.408085 4280 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 21:08:17.408281 4280 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 21:08:17.408540 4280 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 21:08:17.408571 4280 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 21:08:17.408589 4280 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 21:08:17.408631 4280 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 21:08:17.413582 4280 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:37661
I20260924 21:08:17.413615 4408 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:37661 every 8 connection(s)
I20260924 21:08:17.413954 4280 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790284076800039-392-0/minicluster-data/ts-2/data/info.pb
Sep 24 21:08:17 dist-test-slave-c7tk krb5kdc[3905](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284097, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 21:08:17.421433 3990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:17.414565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:49111 (local address 127.0.98.62:41447)
0924 21:08:17.414617 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0924 21:08:17.414619 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 21:08:17.414763 (+ 144us) server_negotiation.cc:407] Connection header received
0924 21:08:17.415473 (+ 710us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:08:17.415473 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:08:17.415482 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:08:17.415524 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:08:17.416020 (+ 496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:17.416457 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:17.417269 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:17.417451 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:17.417530 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:17.419444 (+ 1914us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 21:08:17.419445 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 21:08:17.419445 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 21:08:17.419451 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 21:08:17.420599 (+ 1148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:08:17.420874 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:08:17.420875 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:08:17.420875 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:08:17.420916 (+ 41us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:08:17.421066 (+ 150us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:08:17.421066 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:08:17.421066 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:08:17.421139 (+ 73us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 21:08:17.421176 (+ 37us) server_negotiation.cc:1035] Waiting for connection context
0924 21:08:17.421332 (+ 156us) server_negotiation.cc:299] Negotiation successful
0924 21:08:17.421375 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260924 21:08:17.421664 4411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:17.414545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41447 (local address 127.0.98.3:49111)
0924 21:08:17.414665 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0924 21:08:17.414697 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0924 21:08:17.415406 (+ 709us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 21:08:17.415588 (+ 182us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 21:08:17.415591 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 21:08:17.415796 (+ 205us) client_negotiation.cc:189] Negotiated authn=SASL
0924 21:08:17.415949 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:17.415952 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:17.416519 (+ 567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:17.416520 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:17.417204 (+ 684us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:17.417205 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:17.417536 (+ 331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:17.417536 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:17.417631 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:17.418067 (+ 436us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 21:08:17.418074 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0924 21:08:17.419348 (+ 1274us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 21:08:17.420655 (+ 1307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 21:08:17.420659 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 21:08:17.420660 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 21:08:17.420818 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 21:08:17.420961 (+ 143us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 21:08:17.420961 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 21:08:17.420961 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 21:08:17.421034 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 21:08:17.421178 (+ 144us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 21:08:17.421180 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 21:08:17.421261 (+ 81us) client_negotiation.cc:769] Sending connection context
0924 21:08:17.421323 (+ 62us) client_negotiation.cc:240] Negotiation successful
0924 21:08:17.421412 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":87,"thread_start_us":65,"threads_started":1}
I20260924 21:08:17.422061 4409 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41447
I20260924 21:08:17.422154 4409 heartbeater.cc:464] Registering TS with master...
I20260924 21:08:17.422288 4409 heartbeater.cc:511] Master 127.0.98.62:41447 requested a full tablet report, sending...
I20260924 21:08:17.422611 3949 ts_manager.cc:194] Registered new tserver with Master: 483d31d21fce400881001c0b9cd80abc (127.0.98.3:37661)
I20260924 21:08:17.422978 3949 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:49111
I20260924 21:08:17.423969 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAzNNjS/build/release/bin/kudu as pid 4280
I20260924 21:08:17.424055 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790284076800039-392-0/minicluster-data/ts-2/wal/instance
I20260924 21:08:17.424214 392 external_mini_cluster.cc:1428] Setting key 69b879caf3011937f7ecbf589016fccf
I20260924 21:08:17.424927 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 24 21:08:17 dist-test-slave-c7tk krb5kdc[3905](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284097, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 24 21:08:17 dist-test-slave-c7tk krb5kdc[3905](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284097, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260924 21:08:17.444500 4416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:17.437930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53112 (local address 127.0.98.1:33225)
0924 21:08:17.438084 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0924 21:08:17.438088 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 21:08:17.438096 (+ 8us) server_negotiation.cc:407] Connection header received
0924 21:08:17.438116 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:08:17.438118 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:08:17.438128 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:08:17.438191 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:08:17.438681 (+ 490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:17.439338 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:17.440039 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:17.440304 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:17.440354 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:17.442266 (+ 1912us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 21:08:17.442272 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 21:08:17.442272 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 21:08:17.442278 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 21:08:17.443637 (+ 1359us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:08:17.443905 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:08:17.443906 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:08:17.443907 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:08:17.443953 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:08:17.444093 (+ 140us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:08:17.444093 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:08:17.444093 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:08:17.444186 (+ 93us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 21:08:17.444268 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0924 21:08:17.444332 (+ 64us) server_negotiation.cc:299] Negotiation successful
0924 21:08:17.444388 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":84,"threads_started":1}
Sep 24 21:08:17 dist-test-slave-c7tk krb5kdc[3905](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284097, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 21:08:17.451647 3990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:17.445518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46606 (local address 127.0.98.62:41447)
0924 21:08:17.445561 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0924 21:08:17.445563 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 21:08:17.445571 (+ 8us) server_negotiation.cc:407] Connection header received
0924 21:08:17.445626 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:08:17.445627 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:08:17.445634 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:08:17.445678 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:08:17.446190 (+ 512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:17.446615 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:17.447198 (+ 583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:17.447380 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:17.447448 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:17.449498 (+ 2050us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 21:08:17.449499 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 21:08:17.449500 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 21:08:17.449505 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 21:08:17.450656 (+ 1151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:08:17.450969 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:08:17.450970 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:08:17.450970 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:08:17.451017 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:08:17.451191 (+ 174us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:08:17.451192 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:08:17.451192 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:08:17.451308 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 21:08:17.451401 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0924 21:08:17.451499 (+ 98us) server_negotiation.cc:299] Negotiation successful
0924 21:08:17.451562 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
W20260924 21:08:17.451856 3949 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46606
I20260924 21:08:17.452127 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAzNNjS/build/release/bin/kudu with pid 4012
I20260924 21:08:17.456725 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAzNNjS/build/release/bin/kudu with pid 4146
I20260924 21:08:17.460582 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAzNNjS/build/release/bin/kudu with pid 4280
I20260924 21:08:17.464182 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAzNNjS/build/release/bin/kudu with pid 3918
2026-09-24T21:08:17Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (1608 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790284076800039-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 21:08:17 dist-test-slave-c7tk krb5kdc[4421](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 21:08:17 dist-test-slave-c7tk krb5kdc[4421](info): set up 2 sockets
Sep 24 21:08:17 dist-test-slave-c7tk krb5kdc[4421](info): commencing operation
krb5kdc: starting...
W20260924 21:08:18.389519 392 mini_kdc.cc:121] Time spent starting KDC: real 0.907s user 0.000s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790284076800039-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790284076800039-392-0/krb5kdc/test-user.keytab.
Sep 24 21:08:18 dist-test-slave-c7tk krb5kdc[4421](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284098, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T21:08:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T21:08:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790284076800039-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790284076800039-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 21:08:18.487928 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAzNNjS/build/release/bin/kudu
/tmp/dist-test-taskAzNNjS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790284076800039-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790284076800039-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790284076800039-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790284076800039-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34109
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790284076800039-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42765
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34109
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790284076800039-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790284076800039-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790284076800039-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 21:08:18.556384 4434 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 21:08:18.556574 4434 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 21:08:18.556604 4434 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 21:08:18.557623 4434 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 21:08:18.557678 4434 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 21:08:18.557714 4434 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 21:08:18.557742 4434 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 21:08:18.557768 4434 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 21:08:18.559269 4434 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42765
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790284076800039-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790284076800039-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34109
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790284076800039-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34109
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790284076800039-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4434
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790284076800039-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 19:49:06 UTC on ccycloud.quasar-vhqjoc.root.comops.site
build id 4670864
I20260924 21:08:18.559516 4434 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 21:08:18.559718 4434 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 21:08:18.563722 4439 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 21:08:18.563743 4442 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 21:08:18.563752 4440 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 21:08:18.563836 4434 server_base.cc:1034] running on GCE node
I20260924 21:08:18.564126 4434 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 21:08:18.564368 4434 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 21:08:18.565495 4434 hybrid_clock.cc:648] HybridClock initialized: now 1790284098565484 us; error 27 us; skew 500 ppm
Sep 24 21:08:18 dist-test-slave-c7tk krb5kdc[4421](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284098, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 21:08:18.567510 4434 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 21:08:18.567822 4434 fs_manager.cc:359] Metadata directory not provided
I20260924 21:08:18.567883 4434 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 21:08:18.568017 4434 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 21:08:18.568984 4434 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790284076800039-392-0/minicluster-data/master-0/data/instance:
uuid: "cc607ee1f27447e0a2f6eb8768cb0a2c"
format_stamp: "Formatted at 2026-09-24 21:08:18 on dist-test-slave-c7tk"
server_key: "d3d36d7f9bfa7260e3abc4669d2709ad"
server_key_iv: "1c8b39b6d4b443a9908c57e1680ae8b1"
server_key_version: "encryptionkey@0"
I20260924 21:08:18.569336 4434 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790284076800039-392-0/minicluster-data/master-0/wal/instance:
uuid: "cc607ee1f27447e0a2f6eb8768cb0a2c"
format_stamp: "Formatted at 2026-09-24 21:08:18 on dist-test-slave-c7tk"
server_key: "d3d36d7f9bfa7260e3abc4669d2709ad"
server_key_iv: "1c8b39b6d4b443a9908c57e1680ae8b1"
server_key_version: "encryptionkey@0"
I20260924 21:08:18.570516 4434 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20260924 21:08:18.571091 4448 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 21:08:18.571237 4434 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 21:08:18.571326 4434 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790284076800039-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790284076800039-392-0/minicluster-data/master-0/wal
uuid: "cc607ee1f27447e0a2f6eb8768cb0a2c"
format_stamp: "Formatted at 2026-09-24 21:08:18 on dist-test-slave-c7tk"
server_key: "d3d36d7f9bfa7260e3abc4669d2709ad"
server_key_iv: "1c8b39b6d4b443a9908c57e1680ae8b1"
server_key_version: "encryptionkey@0"
I20260924 21:08:18.571408 4434 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790284076800039-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790284076800039-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790284076800039-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 21:08:18.579742 4434 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 21:08:18.581166 4434 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 21:08:18.581293 4434 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 21:08:18.584738 4434 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34109
I20260924 21:08:18.584808 4500 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34109 every 8 connection(s)
I20260924 21:08:18.585026 4434 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790284076800039-392-0/minicluster-data/master-0/data/info.pb
I20260924 21:08:18.585737 4501 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 21:08:18.587625 4501 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cc607ee1f27447e0a2f6eb8768cb0a2c: Bootstrap starting.
I20260924 21:08:18.588157 4501 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cc607ee1f27447e0a2f6eb8768cb0a2c: Neither blocks nor log segments found. Creating new log.
I20260924 21:08:18.588440 4501 log.cc:824] T 00000000000000000000000000000000 P cc607ee1f27447e0a2f6eb8768cb0a2c: Log is configured to *not* fsync() on all Append() calls
I20260924 21:08:18.589064 4501 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cc607ee1f27447e0a2f6eb8768cb0a2c: No bootstrap required, opened a new log
I20260924 21:08:18.590538 4501 raft_consensus.cc:348] T 00000000000000000000000000000000 P cc607ee1f27447e0a2f6eb8768cb0a2c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc607ee1f27447e0a2f6eb8768cb0a2c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34109 } }
I20260924 21:08:18.590675 4501 raft_consensus.cc:374] T 00000000000000000000000000000000 P cc607ee1f27447e0a2f6eb8768cb0a2c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 21:08:18.590719 4501 raft_consensus.cc:729] T 00000000000000000000000000000000 P cc607ee1f27447e0a2f6eb8768cb0a2c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc607ee1f27447e0a2f6eb8768cb0a2c, State: Initialized, Role: FOLLOWER
I20260924 21:08:18.590850 4501 consensus_queue.cc:260] T 00000000000000000000000000000000 P cc607ee1f27447e0a2f6eb8768cb0a2c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc607ee1f27447e0a2f6eb8768cb0a2c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34109 } }
I20260924 21:08:18.590929 4501 raft_consensus.cc:388] T 00000000000000000000000000000000 P cc607ee1f27447e0a2f6eb8768cb0a2c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 21:08:18.590972 4501 raft_consensus.cc:482] T 00000000000000000000000000000000 P cc607ee1f27447e0a2f6eb8768cb0a2c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 21:08:18.591019 4501 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cc607ee1f27447e0a2f6eb8768cb0a2c [term 0 FOLLOWER]: Advancing to term 1
I20260924 21:08:18.591457 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAzNNjS/build/release/bin/kudu as pid 4434
I20260924 21:08:18.591580 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790284076800039-392-0/minicluster-data/master-0/wal/instance
I20260924 21:08:18.591780 392 external_mini_cluster.cc:1428] Setting key f9f94755b1d0584ac981ee4cb70d2387
I20260924 21:08:18.591995 4501 raft_consensus.cc:504] T 00000000000000000000000000000000 P cc607ee1f27447e0a2f6eb8768cb0a2c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc607ee1f27447e0a2f6eb8768cb0a2c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34109 } }
I20260924 21:08:18.592065 4501 leader_election.cc:302] T 00000000000000000000000000000000 P cc607ee1f27447e0a2f6eb8768cb0a2c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cc607ee1f27447e0a2f6eb8768cb0a2c; no voters:
I20260924 21:08:18.592207 4501 leader_election.cc:288] T 00000000000000000000000000000000 P cc607ee1f27447e0a2f6eb8768cb0a2c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 21:08:18.592305 4504 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cc607ee1f27447e0a2f6eb8768cb0a2c [term 1 FOLLOWER]: Leader election won for term 1
I20260924 21:08:18.592424 4501 sys_catalog.cc:564] T 00000000000000000000000000000000 P cc607ee1f27447e0a2f6eb8768cb0a2c [sys.catalog]: configured and running, proceeding with master startup.
I20260924 21:08:18.592429 4504 raft_consensus.cc:686] T 00000000000000000000000000000000 P cc607ee1f27447e0a2f6eb8768cb0a2c [term 1 LEADER]: Becoming Leader. State: Replica: cc607ee1f27447e0a2f6eb8768cb0a2c, State: Running, Role: LEADER
I20260924 21:08:18.592614 4504 consensus_queue.cc:237] T 00000000000000000000000000000000 P cc607ee1f27447e0a2f6eb8768cb0a2c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc607ee1f27447e0a2f6eb8768cb0a2c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34109 } }
I20260924 21:08:18.592979 4508 sys_catalog.cc:455] T 00000000000000000000000000000000 P cc607ee1f27447e0a2f6eb8768cb0a2c [sys.catalog]: SysCatalogTable state changed. Reason: New leader cc607ee1f27447e0a2f6eb8768cb0a2c. Latest consensus state: current_term: 1 leader_uuid: "cc607ee1f27447e0a2f6eb8768cb0a2c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc607ee1f27447e0a2f6eb8768cb0a2c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34109 } } }
I20260924 21:08:18.593070 4508 sys_catalog.cc:458] T 00000000000000000000000000000000 P cc607ee1f27447e0a2f6eb8768cb0a2c [sys.catalog]: This master's current role is: LEADER
I20260924 21:08:18.592996 4507 sys_catalog.cc:455] T 00000000000000000000000000000000 P cc607ee1f27447e0a2f6eb8768cb0a2c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cc607ee1f27447e0a2f6eb8768cb0a2c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc607ee1f27447e0a2f6eb8768cb0a2c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34109 } } }
I20260924 21:08:18.593254 4507 sys_catalog.cc:458] T 00000000000000000000000000000000 P cc607ee1f27447e0a2f6eb8768cb0a2c [sys.catalog]: This master's current role is: LEADER
I20260924 21:08:18.593246 4515 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 21:08:18.593899 4515 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 24 21:08:18 dist-test-slave-c7tk krb5kdc[4421](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284098, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 21:08:18.598670 4515 catalog_manager.cc:1347] Generated new cluster ID: 4a4b910dd54746dbab76d7fb374b452e
I20260924 21:08:18.598726 4515 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 21:08:18.601946 4506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:18.592206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49234 (local address 127.0.98.62:34109)
0924 21:08:18.592365 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0924 21:08:18.592373 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 21:08:18.592557 (+ 184us) server_negotiation.cc:407] Connection header received
0924 21:08:18.592775 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:08:18.592781 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:08:18.592826 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:08:18.593706 (+ 880us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:08:18.593883 (+ 177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:18.595019 (+ 1136us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:18.595736 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:18.596102 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:18.596205 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:18.598623 (+ 2418us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 21:08:18.598625 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 21:08:18.598626 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 21:08:18.598634 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 21:08:18.600601 (+ 1967us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:08:18.600945 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:08:18.600948 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:08:18.600948 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:08:18.601023 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:08:18.601205 (+ 182us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:08:18.601206 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:08:18.601206 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:08:18.601387 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 21:08:18.601551 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0924 21:08:18.601632 (+ 81us) server_negotiation.cc:299] Negotiation successful
0924 21:08:18.601751 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116,"thread_start_us":60,"threads_started":1}
I20260924 21:08:18.618511 4515 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 21:08:18.619093 4515 catalog_manager.cc:1504] Loading token signing keys...
I20260924 21:08:18.622372 4515 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cc607ee1f27447e0a2f6eb8768cb0a2c: Generated new TSK 0
I20260924 21:08:18.622514 4515 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790284076800039-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790284076800039-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260924 21:08:18.647660 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAzNNjS/build/release/bin/kudu
/tmp/dist-test-taskAzNNjS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790284076800039-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790284076800039-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790284076800039-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790284076800039-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34109
--keytab_file=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790284076800039-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42765
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790284076800039-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790284076800039-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790284076800039-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 21:08:18.734997 4527 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 21:08:18.735152 4527 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 21:08:18.735172 4527 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 21:08:18.736119 4527 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 21:08:18.736155 4527 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 21:08:18.736177 4527 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 21:08:18.737469 4527 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42765
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790284076800039-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790284076800039-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790284076800039-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790284076800039-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34109
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4527
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790284076800039-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 19:49:06 UTC on ccycloud.quasar-vhqjoc.root.comops.site
build id 4670864
I20260924 21:08:18.737610 4527 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 21:08:18.737866 4527 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 21:08:18.742143 4533 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 21:08:18.742177 4535 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 21:08:18.742204 4527 server_base.cc:1034] running on GCE node
W20260924 21:08:18.742141 4532 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 21:08:18.742461 4527 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 21:08:18.742676 4527 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 21:08:18.743811 4527 hybrid_clock.cc:648] HybridClock initialized: now 1790284098743799 us; error 27 us; skew 500 ppm
Sep 24 21:08:18 dist-test-slave-c7tk krb5kdc[4421](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284098, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 21:08:18.745918 4527 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 21:08:18.746186 4527 fs_manager.cc:359] Metadata directory not provided
I20260924 21:08:18.746225 4527 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 21:08:18.746340 4527 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 21:08:18.747159 4527 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790284076800039-392-0/minicluster-data/ts-0/data/instance:
uuid: "913df449ff944b7fb8e229b6a7ae1867"
format_stamp: "Formatted at 2026-09-24 21:08:18 on dist-test-slave-c7tk"
server_key: "fc37e10657b4688078f5dda8445bd5e0"
server_key_iv: "c73d4dcea9577f83bfe2c5f6da4dd3a7"
server_key_version: "encryptionkey@0"
I20260924 21:08:18.747481 4527 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790284076800039-392-0/minicluster-data/ts-0/wal/instance:
uuid: "913df449ff944b7fb8e229b6a7ae1867"
format_stamp: "Formatted at 2026-09-24 21:08:18 on dist-test-slave-c7tk"
server_key: "fc37e10657b4688078f5dda8445bd5e0"
server_key_iv: "c73d4dcea9577f83bfe2c5f6da4dd3a7"
server_key_version: "encryptionkey@0"
I20260924 21:08:18.748595 4527 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 21:08:18.749257 4541 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 21:08:18.749399 4527 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 21:08:18.749478 4527 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790284076800039-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790284076800039-392-0/minicluster-data/ts-0/wal
uuid: "913df449ff944b7fb8e229b6a7ae1867"
format_stamp: "Formatted at 2026-09-24 21:08:18 on dist-test-slave-c7tk"
server_key: "fc37e10657b4688078f5dda8445bd5e0"
server_key_iv: "c73d4dcea9577f83bfe2c5f6da4dd3a7"
server_key_version: "encryptionkey@0"
I20260924 21:08:18.749548 4527 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790284076800039-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790284076800039-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790284076800039-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 21:08:18.759527 4527 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 21:08:18.761083 4527 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 21:08:18.761209 4527 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 21:08:18.761384 4527 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 21:08:18.761641 4527 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 21:08:18.761683 4527 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 21:08:18.761713 4527 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 21:08:18.761739 4527 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 21:08:18.766855 4527 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:35919
I20260924 21:08:18.766913 4654 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:35919 every 8 connection(s)
I20260924 21:08:18.767107 4527 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790284076800039-392-0/minicluster-data/ts-0/data/info.pb
Sep 24 21:08:18 dist-test-slave-c7tk krb5kdc[4421](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284098, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 21:08:18.772060 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAzNNjS/build/release/bin/kudu as pid 4527
I20260924 21:08:18.772167 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790284076800039-392-0/minicluster-data/ts-0/wal/instance
I20260924 21:08:18.772330 392 external_mini_cluster.cc:1428] Setting key d61dcb2c7d9e42aa52dff7826e71ffca
I20260924 21:08:18.774899 4657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:18.767632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34109 (local address 127.0.98.1:46935)
0924 21:08:18.767765 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0924 21:08:18.767795 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0924 21:08:18.768185 (+ 390us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 21:08:18.768363 (+ 178us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 21:08:18.768365 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 21:08:18.768591 (+ 226us) client_negotiation.cc:189] Negotiated authn=SASL
0924 21:08:18.769019 (+ 428us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:18.769023 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:18.769636 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:18.769637 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:18.770339 (+ 702us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:18.770341 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:18.770621 (+ 280us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:18.770622 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:18.770697 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:18.771013 (+ 316us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 21:08:18.771019 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0924 21:08:18.772189 (+ 1170us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 21:08:18.773642 (+ 1453us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 21:08:18.773648 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 21:08:18.773649 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 21:08:18.773956 (+ 307us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 21:08:18.774149 (+ 193us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 21:08:18.774150 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 21:08:18.774150 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 21:08:18.774227 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 21:08:18.774457 (+ 230us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 21:08:18.774461 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 21:08:18.774603 (+ 142us) client_negotiation.cc:769] Sending connection context
0924 21:08:18.774674 (+ 71us) client_negotiation.cc:240] Negotiation successful
0924 21:08:18.774768 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":75,"threads_started":1}
I20260924 21:08:18.774910 4506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:18.767666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:46935 (local address 127.0.98.62:34109)
0924 21:08:18.767698 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0924 21:08:18.767700 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 21:08:18.767853 (+ 153us) server_negotiation.cc:407] Connection header received
0924 21:08:18.768258 (+ 405us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:08:18.768258 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:08:18.768267 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:08:18.768319 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:08:18.769088 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:18.769576 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:18.770410 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:18.770568 (+ 158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:18.770610 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:18.772310 (+ 1700us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 21:08:18.772312 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 21:08:18.772312 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 21:08:18.772318 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 21:08:18.773564 (+ 1246us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:08:18.774029 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:08:18.774030 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:08:18.774031 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:08:18.774091 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:08:18.774275 (+ 184us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:08:18.774275 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:08:18.774276 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:08:18.774398 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 21:08:18.774469 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0924 21:08:18.774708 (+ 239us) server_negotiation.cc:299] Negotiation successful
0924 21:08:18.774806 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14}
I20260924 21:08:18.775251 4655 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34109
I20260924 21:08:18.775321 4655 heartbeater.cc:464] Registering TS with master...
I20260924 21:08:18.775439 4655 heartbeater.cc:511] Master 127.0.98.62:34109 requested a full tablet report, sending...
I20260924 21:08:18.775846 4465 ts_manager.cc:194] Registered new tserver with Master: 913df449ff944b7fb8e229b6a7ae1867 (127.0.98.1:35919)
I20260924 21:08:18.776337 4465 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:46935
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790284076800039-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790284076800039-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260924 21:08:18.793918 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAzNNjS/build/release/bin/kudu
/tmp/dist-test-taskAzNNjS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790284076800039-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790284076800039-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790284076800039-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790284076800039-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34109
--keytab_file=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790284076800039-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42765
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790284076800039-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790284076800039-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790284076800039-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 21:08:18.861634 4660 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 21:08:18.861778 4660 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 21:08:18.861810 4660 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 21:08:18.862767 4660 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 21:08:18.862802 4660 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 21:08:18.862843 4660 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 21:08:18.864080 4660 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42765
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790284076800039-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790284076800039-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790284076800039-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790284076800039-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34109
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4660
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790284076800039-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 19:49:06 UTC on ccycloud.quasar-vhqjoc.root.comops.site
build id 4670864
I20260924 21:08:18.864218 4660 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 21:08:18.864411 4660 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 21:08:18.868463 4668 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 21:08:18.868573 4660 server_base.cc:1034] running on GCE node
W20260924 21:08:18.868467 4666 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 21:08:18.868481 4665 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 21:08:18.868899 4660 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 21:08:18.869108 4660 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 21:08:18.870251 4660 hybrid_clock.cc:648] HybridClock initialized: now 1790284098870234 us; error 28 us; skew 500 ppm
Sep 24 21:08:18 dist-test-slave-c7tk krb5kdc[4421](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284098, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 21:08:18.872359 4660 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 21:08:18.872648 4660 fs_manager.cc:359] Metadata directory not provided
I20260924 21:08:18.872706 4660 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 21:08:18.872817 4660 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 21:08:18.873661 4660 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790284076800039-392-0/minicluster-data/ts-1/data/instance:
uuid: "f11892e20695415d969c83d2ca192871"
format_stamp: "Formatted at 2026-09-24 21:08:18 on dist-test-slave-c7tk"
server_key: "e9d0ff68a9970c74fddc441d1871fd66"
server_key_iv: "85243586c676d94e823949fcff04cd59"
server_key_version: "encryptionkey@0"
I20260924 21:08:18.874009 4660 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790284076800039-392-0/minicluster-data/ts-1/wal/instance:
uuid: "f11892e20695415d969c83d2ca192871"
format_stamp: "Formatted at 2026-09-24 21:08:18 on dist-test-slave-c7tk"
server_key: "e9d0ff68a9970c74fddc441d1871fd66"
server_key_iv: "85243586c676d94e823949fcff04cd59"
server_key_version: "encryptionkey@0"
I20260924 21:08:18.875133 4660 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 21:08:18.875698 4674 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 21:08:18.875831 4660 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 21:08:18.875908 4660 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790284076800039-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790284076800039-392-0/minicluster-data/ts-1/wal
uuid: "f11892e20695415d969c83d2ca192871"
format_stamp: "Formatted at 2026-09-24 21:08:18 on dist-test-slave-c7tk"
server_key: "e9d0ff68a9970c74fddc441d1871fd66"
server_key_iv: "85243586c676d94e823949fcff04cd59"
server_key_version: "encryptionkey@0"
I20260924 21:08:18.875975 4660 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790284076800039-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790284076800039-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790284076800039-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 21:08:18.884416 4660 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 21:08:18.886575 4660 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 21:08:18.886736 4660 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 21:08:18.886942 4660 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 21:08:18.887234 4660 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 21:08:18.887285 4660 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 21:08:18.887344 4660 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 21:08:18.887377 4660 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 21:08:18.893256 4660 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:43007
I20260924 21:08:18.893316 4787 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:43007 every 8 connection(s)
I20260924 21:08:18.893561 4660 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790284076800039-392-0/minicluster-data/ts-1/data/info.pb
I20260924 21:08:18.897286 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAzNNjS/build/release/bin/kudu as pid 4660
I20260924 21:08:18.897399 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790284076800039-392-0/minicluster-data/ts-1/wal/instance
I20260924 21:08:18.897612 392 external_mini_cluster.cc:1428] Setting key c3fad54283bd265ed7f66e37325bd74c
Sep 24 21:08:18 dist-test-slave-c7tk krb5kdc[4421](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284098, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 21:08:18.901638 4506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:18.894229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:60165 (local address 127.0.98.62:34109)
0924 21:08:18.894277 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0924 21:08:18.894280 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 21:08:18.894472 (+ 192us) server_negotiation.cc:407] Connection header received
0924 21:08:18.894867 (+ 395us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:08:18.894868 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:08:18.894877 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:08:18.894929 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:08:18.895622 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:18.896048 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:18.896946 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:18.897129 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:18.897190 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:18.899318 (+ 2128us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 21:08:18.899319 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 21:08:18.899320 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 21:08:18.899326 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 21:08:18.900524 (+ 1198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:08:18.900855 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:08:18.900856 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:08:18.900857 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:08:18.900911 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:08:18.901110 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:08:18.901111 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:08:18.901111 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:08:18.901206 (+ 95us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 21:08:18.901268 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0924 21:08:18.901489 (+ 221us) server_negotiation.cc:299] Negotiation successful
0924 21:08:18.901562 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260924 21:08:18.901755 4790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:18.894216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34109 (local address 127.0.98.2:60165)
0924 21:08:18.894357 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0924 21:08:18.894387 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0924 21:08:18.894784 (+ 397us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 21:08:18.894969 (+ 185us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 21:08:18.894971 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 21:08:18.895166 (+ 195us) client_negotiation.cc:189] Negotiated authn=SASL
0924 21:08:18.895557 (+ 391us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:18.895560 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:18.896114 (+ 554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:18.896114 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:18.896846 (+ 732us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:18.896848 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:18.897210 (+ 362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:18.897211 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:18.897357 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:18.897820 (+ 463us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 21:08:18.897829 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 21:08:18.899195 (+ 1366us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 21:08:18.900609 (+ 1414us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 21:08:18.900614 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 21:08:18.900616 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 21:08:18.900793 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 21:08:18.900964 (+ 171us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 21:08:18.900965 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 21:08:18.900965 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 21:08:18.901056 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 21:08:18.901260 (+ 204us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 21:08:18.901264 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 21:08:18.901375 (+ 111us) client_negotiation.cc:769] Sending connection context
0924 21:08:18.901460 (+ 85us) client_negotiation.cc:240] Negotiation successful
0924 21:08:18.901579 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":81,"threads_started":1}
I20260924 21:08:18.902100 4788 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34109
I20260924 21:08:18.902172 4788 heartbeater.cc:464] Registering TS with master...
I20260924 21:08:18.902302 4788 heartbeater.cc:511] Master 127.0.98.62:34109 requested a full tablet report, sending...
I20260924 21:08:18.902570 4465 ts_manager.cc:194] Registered new tserver with Master: f11892e20695415d969c83d2ca192871 (127.0.98.2:43007)
I20260924 21:08:18.902884 4465 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:60165
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790284076800039-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790284076800039-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260924 21:08:18.918337 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAzNNjS/build/release/bin/kudu
/tmp/dist-test-taskAzNNjS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790284076800039-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790284076800039-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790284076800039-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790284076800039-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34109
--keytab_file=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790284076800039-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42765
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790284076800039-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790284076800039-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790284076800039-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 21:08:18.986872 4793 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 21:08:18.987030 4793 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 21:08:18.987048 4793 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 21:08:18.987977 4793 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 21:08:18.988024 4793 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 21:08:18.988050 4793 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 21:08:18.989409 4793 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42765
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790284076800039-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790284076800039-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790284076800039-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790284076800039-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34109
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4793
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790284076800039-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 19:49:06 UTC on ccycloud.quasar-vhqjoc.root.comops.site
build id 4670864
I20260924 21:08:18.989547 4793 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 21:08:18.989679 4793 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 21:08:18.994021 4801 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 21:08:18.994064 4798 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 21:08:18.994206 4793 server_base.cc:1034] running on GCE node
W20260924 21:08:18.994220 4799 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 21:08:18.994405 4793 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 21:08:18.994638 4793 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 21:08:18.995774 4793 hybrid_clock.cc:648] HybridClock initialized: now 1790284098995765 us; error 22 us; skew 500 ppm
Sep 24 21:08:18 dist-test-slave-c7tk krb5kdc[4421](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284098, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 21:08:18.997843 4793 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 21:08:18.998160 4793 fs_manager.cc:359] Metadata directory not provided
I20260924 21:08:18.998224 4793 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 21:08:18.998356 4793 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 21:08:18.999305 4793 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790284076800039-392-0/minicluster-data/ts-2/data/instance:
uuid: "77f3eb0b84f84d81873b63de3df030b5"
format_stamp: "Formatted at 2026-09-24 21:08:18 on dist-test-slave-c7tk"
server_key: "621116c1d5fb77f390819a07751fd61d"
server_key_iv: "61c712623a088f32216d56d1cab965eb"
server_key_version: "encryptionkey@0"
I20260924 21:08:18.999591 4793 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790284076800039-392-0/minicluster-data/ts-2/wal/instance:
uuid: "77f3eb0b84f84d81873b63de3df030b5"
format_stamp: "Formatted at 2026-09-24 21:08:18 on dist-test-slave-c7tk"
server_key: "621116c1d5fb77f390819a07751fd61d"
server_key_iv: "61c712623a088f32216d56d1cab965eb"
server_key_version: "encryptionkey@0"
I20260924 21:08:19.000590 4793 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 21:08:19.001113 4807 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 21:08:19.001235 4793 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 21:08:19.001312 4793 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790284076800039-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790284076800039-392-0/minicluster-data/ts-2/wal
uuid: "77f3eb0b84f84d81873b63de3df030b5"
format_stamp: "Formatted at 2026-09-24 21:08:18 on dist-test-slave-c7tk"
server_key: "621116c1d5fb77f390819a07751fd61d"
server_key_iv: "61c712623a088f32216d56d1cab965eb"
server_key_version: "encryptionkey@0"
I20260924 21:08:19.001374 4793 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790284076800039-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790284076800039-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790284076800039-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 21:08:19.011829 4793 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 21:08:19.013630 4793 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 21:08:19.013783 4793 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 21:08:19.014072 4793 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 21:08:19.014412 4793 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 21:08:19.014456 4793 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 21:08:19.014492 4793 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 21:08:19.014524 4793 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 21:08:19.019883 4793 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:40601
I20260924 21:08:19.019943 4920 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:40601 every 8 connection(s)
I20260924 21:08:19.020162 4793 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790284076800039-392-0/minicluster-data/ts-2/data/info.pb
I20260924 21:08:19.021800 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAzNNjS/build/release/bin/kudu as pid 4793
I20260924 21:08:19.021953 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790284076800039-392-0/minicluster-data/ts-2/wal/instance
I20260924 21:08:19.022162 392 external_mini_cluster.cc:1428] Setting key 483b3cebffd15dd9baabb02d5f35fc37
Sep 24 21:08:19 dist-test-slave-c7tk krb5kdc[4421](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284098, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 21:08:19.028738 4506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:19.020853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:35803 (local address 127.0.98.62:34109)
0924 21:08:19.020914 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0924 21:08:19.020916 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 21:08:19.021144 (+ 228us) server_negotiation.cc:407] Connection header received
0924 21:08:19.021517 (+ 373us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:08:19.021518 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:08:19.021530 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:08:19.021601 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:08:19.022891 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:19.023370 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:19.024471 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:19.024641 (+ 170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:19.024692 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:19.026618 (+ 1926us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 21:08:19.026619 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 21:08:19.026619 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 21:08:19.026625 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 21:08:19.027683 (+ 1058us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:08:19.028038 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:08:19.028039 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:08:19.028039 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:08:19.028084 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:08:19.028281 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:08:19.028282 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:08:19.028282 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:08:19.028372 (+ 90us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 21:08:19.028419 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0924 21:08:19.028623 (+ 204us) server_negotiation.cc:299] Negotiation successful
0924 21:08:19.028678 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260924 21:08:19.028954 4923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:19.020860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34109 (local address 127.0.98.3:35803)
0924 21:08:19.021016 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0924 21:08:19.021043 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0924 21:08:19.021425 (+ 382us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 21:08:19.021725 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 21:08:19.021730 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 21:08:19.022120 (+ 390us) client_negotiation.cc:189] Negotiated authn=SASL
0924 21:08:19.022691 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:19.022697 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:19.023442 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:19.023443 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:19.024393 (+ 950us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:19.024395 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:19.024707 (+ 312us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:19.024708 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:19.024814 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:19.025235 (+ 421us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 21:08:19.025243 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 21:08:19.026543 (+ 1300us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 21:08:19.027743 (+ 1200us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 21:08:19.027748 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 21:08:19.027749 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 21:08:19.027956 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 21:08:19.028139 (+ 183us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 21:08:19.028139 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 21:08:19.028139 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 21:08:19.028230 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 21:08:19.028422 (+ 192us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 21:08:19.028426 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 21:08:19.028522 (+ 96us) client_negotiation.cc:769] Sending connection context
0924 21:08:19.028602 (+ 80us) client_negotiation.cc:240] Negotiation successful
0924 21:08:19.028707 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":96,"threads_started":1}
I20260924 21:08:19.029420 4921 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34109
I20260924 21:08:19.029508 4921 heartbeater.cc:464] Registering TS with master...
I20260924 21:08:19.029636 4921 heartbeater.cc:511] Master 127.0.98.62:34109 requested a full tablet report, sending...
I20260924 21:08:19.029994 4465 ts_manager.cc:194] Registered new tserver with Master: 77f3eb0b84f84d81873b63de3df030b5 (127.0.98.3:40601)
I20260924 21:08:19.030316 4465 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:35803
I20260924 21:08:19.033671 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 21:08:19.039649 4506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:19.034888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49246 (local address 127.0.98.62:34109)
0924 21:08:19.034938 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0924 21:08:19.034939 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 21:08:19.035007 (+ 68us) server_negotiation.cc:407] Connection header received
0924 21:08:19.035060 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:08:19.035060 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:08:19.035067 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:08:19.035108 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:08:19.035541 (+ 433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:19.035914 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:19.036553 (+ 639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:19.036702 (+ 149us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:19.036755 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:19.037583 (+ 828us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 21:08:19.037583 (+ 0us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 21:08:19.037583 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 21:08:19.037587 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 21:08:19.038869 (+ 1282us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:08:19.039139 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:08:19.039140 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:08:19.039140 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:08:19.039182 (+ 42us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:08:19.039325 (+ 143us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:08:19.039325 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:08:19.039325 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:08:19.039424 (+ 99us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 21:08:19.039482 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0924 21:08:19.039546 (+ 64us) server_negotiation.cc:299] Negotiation successful
0924 21:08:19.039588 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260924 21:08:19.040930 4465 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49246:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 21:08:19.042244 4465 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 21:08:19.047303 4933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:19.044085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43007 (local address 127.0.0.1:49800)
0924 21:08:19.044208 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0924 21:08:19.044216 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0924 21:08:19.044249 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 21:08:19.044623 (+ 374us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 21:08:19.044623 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 21:08:19.044625 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 21:08:19.044680 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:19.044681 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:19.045656 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:19.045656 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:19.046336 (+ 680us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:19.046338 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:19.047070 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:19.047071 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:19.047134 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:19.047146 (+ 12us) client_negotiation.cc:769] Sending connection context
0924 21:08:19.047205 (+ 59us) client_negotiation.cc:240] Negotiation successful
0924 21:08:19.047246 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":60,"threads_started":1}
I20260924 21:08:19.047314 4934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:19.044158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49800 (local address 127.0.98.2:43007)
0924 21:08:19.044423 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0924 21:08:19.044427 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 21:08:19.044435 (+ 8us) server_negotiation.cc:407] Connection header received
0924 21:08:19.044466 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:08:19.044469 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:08:19.044482 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:08:19.044548 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 21:08:19.044747 (+ 199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:19.045574 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:19.046425 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:19.046989 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:19.047067 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:19.047113 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0924 21:08:19.047215 (+ 102us) server_negotiation.cc:299] Negotiation successful
0924 21:08:19.047249 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":226,"thread_start_us":76,"threads_started":1}
I20260924 21:08:19.047569 4929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:19.043890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40601 (local address 127.0.0.1:47640)
0924 21:08:19.044029 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0924 21:08:19.044059 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0924 21:08:19.044109 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 21:08:19.044322 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 21:08:19.044325 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 21:08:19.044326 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 21:08:19.044375 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:19.044376 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:19.045386 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:19.045387 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:19.046530 (+ 1143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:19.046532 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:19.047329 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:19.047330 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:19.047387 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:19.047390 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 21:08:19.047475 (+ 85us) client_negotiation.cc:240] Negotiation successful
0924 21:08:19.047508 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"mutex_wait_us":71,"thread_start_us":55,"threads_started":1}
I20260924 21:08:19.047588 4931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:19.043946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47640 (local address 127.0.98.3:40601)
0924 21:08:19.044178 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0924 21:08:19.044182 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 21:08:19.044192 (+ 10us) server_negotiation.cc:407] Connection header received
0924 21:08:19.044230 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:08:19.044232 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:08:19.044244 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:08:19.044317 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 21:08:19.044440 (+ 123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:19.045297 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:19.046621 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:19.047193 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:19.047233 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:19.047277 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0924 21:08:19.047494 (+ 217us) server_negotiation.cc:299] Negotiation successful
0924 21:08:19.047528 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":209,"thread_start_us":67,"threads_started":1}
I20260924 21:08:19.047879 4722 tablet_service.cc:1467] Processing CreateTablet for tablet 859ea66ebf114eaf86bc62248d768c92 (DEFAULT_TABLE table=test-table [id=2bd494b552fe4a5e9191a9b53c23bb0a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 21:08:19.047979 4930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:19.043890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35919 (local address 127.0.0.1:41664)
0924 21:08:19.044190 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0924 21:08:19.044198 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0924 21:08:19.044236 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 21:08:19.044481 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 21:08:19.044481 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 21:08:19.044483 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 21:08:19.044512 (+ 29us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:19.044512 (+ 0us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:19.045351 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:19.045352 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:19.046538 (+ 1186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:19.046540 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:19.047578 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:19.047578 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:19.047837 (+ 259us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:19.047841 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 21:08:19.047882 (+ 41us) client_negotiation.cc:240] Negotiation successful
0924 21:08:19.047921 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":285,"mutex_wait_us":74,"thread_start_us":62,"threads_started":1}
I20260924 21:08:19.048197 4722 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 859ea66ebf114eaf86bc62248d768c92. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 21:08:19.048138 4855 tablet_service.cc:1467] Processing CreateTablet for tablet 859ea66ebf114eaf86bc62248d768c92 (DEFAULT_TABLE table=test-table [id=2bd494b552fe4a5e9191a9b53c23bb0a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 21:08:19.048288 4932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:19.044041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41664 (local address 127.0.98.1:35919)
0924 21:08:19.044346 (+ 305us) server_negotiation.cc:206] Beginning negotiation
0924 21:08:19.044350 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 21:08:19.044357 (+ 7us) server_negotiation.cc:407] Connection header received
0924 21:08:19.044390 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:08:19.044401 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:08:19.044416 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:08:19.044483 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 21:08:19.044559 (+ 76us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:19.045271 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:19.046617 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:19.047251 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:19.047303 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:19.048146 (+ 843us) server_negotiation.cc:1035] Waiting for connection context
0924 21:08:19.048174 (+ 28us) server_negotiation.cc:299] Negotiation successful
0924 21:08:19.048216 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":275,"thread_start_us":58,"threads_started":1}
I20260924 21:08:19.048422 4855 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 859ea66ebf114eaf86bc62248d768c92. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 21:08:19.050592 4935 tablet_bootstrap.cc:492] T 859ea66ebf114eaf86bc62248d768c92 P 77f3eb0b84f84d81873b63de3df030b5: Bootstrap starting.
I20260924 21:08:19.050627 4936 tablet_bootstrap.cc:492] T 859ea66ebf114eaf86bc62248d768c92 P f11892e20695415d969c83d2ca192871: Bootstrap starting.
I20260924 21:08:19.051229 4935 tablet_bootstrap.cc:654] T 859ea66ebf114eaf86bc62248d768c92 P 77f3eb0b84f84d81873b63de3df030b5: Neither blocks nor log segments found. Creating new log.
I20260924 21:08:19.051251 4936 tablet_bootstrap.cc:654] T 859ea66ebf114eaf86bc62248d768c92 P f11892e20695415d969c83d2ca192871: Neither blocks nor log segments found. Creating new log.
I20260924 21:08:19.051196 4589 tablet_service.cc:1467] Processing CreateTablet for tablet 859ea66ebf114eaf86bc62248d768c92 (DEFAULT_TABLE table=test-table [id=2bd494b552fe4a5e9191a9b53c23bb0a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 21:08:19.051551 4589 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 859ea66ebf114eaf86bc62248d768c92. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 21:08:19.051816 4935 log.cc:824] T 859ea66ebf114eaf86bc62248d768c92 P 77f3eb0b84f84d81873b63de3df030b5: Log is configured to *not* fsync() on all Append() calls
I20260924 21:08:19.051909 4936 log.cc:824] T 859ea66ebf114eaf86bc62248d768c92 P f11892e20695415d969c83d2ca192871: Log is configured to *not* fsync() on all Append() calls
I20260924 21:08:19.052480 4935 tablet_bootstrap.cc:492] T 859ea66ebf114eaf86bc62248d768c92 P 77f3eb0b84f84d81873b63de3df030b5: No bootstrap required, opened a new log
I20260924 21:08:19.052547 4935 ts_tablet_manager.cc:1397] T 859ea66ebf114eaf86bc62248d768c92 P 77f3eb0b84f84d81873b63de3df030b5: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 21:08:19.052610 4936 tablet_bootstrap.cc:492] T 859ea66ebf114eaf86bc62248d768c92 P f11892e20695415d969c83d2ca192871: No bootstrap required, opened a new log
I20260924 21:08:19.052672 4936 ts_tablet_manager.cc:1397] T 859ea66ebf114eaf86bc62248d768c92 P f11892e20695415d969c83d2ca192871: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 21:08:19.053779 4935 raft_consensus.cc:348] T 859ea66ebf114eaf86bc62248d768c92 P 77f3eb0b84f84d81873b63de3df030b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "913df449ff944b7fb8e229b6a7ae1867" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35919 } } peers { permanent_uuid: "77f3eb0b84f84d81873b63de3df030b5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40601 } } peers { permanent_uuid: "f11892e20695415d969c83d2ca192871" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43007 } }
I20260924 21:08:19.053947 4935 raft_consensus.cc:374] T 859ea66ebf114eaf86bc62248d768c92 P 77f3eb0b84f84d81873b63de3df030b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 21:08:19.053985 4935 raft_consensus.cc:729] T 859ea66ebf114eaf86bc62248d768c92 P 77f3eb0b84f84d81873b63de3df030b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77f3eb0b84f84d81873b63de3df030b5, State: Initialized, Role: FOLLOWER
I20260924 21:08:19.054092 4935 consensus_queue.cc:260] T 859ea66ebf114eaf86bc62248d768c92 P 77f3eb0b84f84d81873b63de3df030b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "913df449ff944b7fb8e229b6a7ae1867" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35919 } } peers { permanent_uuid: "77f3eb0b84f84d81873b63de3df030b5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40601 } } peers { permanent_uuid: "f11892e20695415d969c83d2ca192871" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43007 } }
I20260924 21:08:19.054317 4935 ts_tablet_manager.cc:1428] T 859ea66ebf114eaf86bc62248d768c92 P 77f3eb0b84f84d81873b63de3df030b5: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260924 21:08:19.054426 4921 heartbeater.cc:503] Master 127.0.98.62:34109 was elected leader, sending a full tablet report...
I20260924 21:08:19.054443 4936 raft_consensus.cc:348] T 859ea66ebf114eaf86bc62248d768c92 P f11892e20695415d969c83d2ca192871 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "913df449ff944b7fb8e229b6a7ae1867" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35919 } } peers { permanent_uuid: "77f3eb0b84f84d81873b63de3df030b5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40601 } } peers { permanent_uuid: "f11892e20695415d969c83d2ca192871" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43007 } }
I20260924 21:08:19.054555 4940 tablet_bootstrap.cc:492] T 859ea66ebf114eaf86bc62248d768c92 P 913df449ff944b7fb8e229b6a7ae1867: Bootstrap starting.
I20260924 21:08:19.054590 4936 raft_consensus.cc:374] T 859ea66ebf114eaf86bc62248d768c92 P f11892e20695415d969c83d2ca192871 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 21:08:19.054629 4936 raft_consensus.cc:729] T 859ea66ebf114eaf86bc62248d768c92 P f11892e20695415d969c83d2ca192871 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f11892e20695415d969c83d2ca192871, State: Initialized, Role: FOLLOWER
I20260924 21:08:19.054793 4936 consensus_queue.cc:260] T 859ea66ebf114eaf86bc62248d768c92 P f11892e20695415d969c83d2ca192871 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "913df449ff944b7fb8e229b6a7ae1867" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35919 } } peers { permanent_uuid: "77f3eb0b84f84d81873b63de3df030b5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40601 } } peers { permanent_uuid: "f11892e20695415d969c83d2ca192871" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43007 } }
I20260924 21:08:19.054939 4941 raft_consensus.cc:482] T 859ea66ebf114eaf86bc62248d768c92 P f11892e20695415d969c83d2ca192871 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 21:08:19.055047 4936 ts_tablet_manager.cc:1428] T 859ea66ebf114eaf86bc62248d768c92 P f11892e20695415d969c83d2ca192871: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260924 21:08:19.055085 4940 tablet_bootstrap.cc:654] T 859ea66ebf114eaf86bc62248d768c92 P 913df449ff944b7fb8e229b6a7ae1867: Neither blocks nor log segments found. Creating new log.
I20260924 21:08:19.055200 4788 heartbeater.cc:503] Master 127.0.98.62:34109 was elected leader, sending a full tablet report...
I20260924 21:08:19.055047 4941 raft_consensus.cc:504] T 859ea66ebf114eaf86bc62248d768c92 P f11892e20695415d969c83d2ca192871 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "913df449ff944b7fb8e229b6a7ae1867" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35919 } } peers { permanent_uuid: "77f3eb0b84f84d81873b63de3df030b5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40601 } } peers { permanent_uuid: "f11892e20695415d969c83d2ca192871" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43007 } }
I20260924 21:08:19.055630 4940 log.cc:824] T 859ea66ebf114eaf86bc62248d768c92 P 913df449ff944b7fb8e229b6a7ae1867: Log is configured to *not* fsync() on all Append() calls
I20260924 21:08:19.055734 4941 leader_election.cc:288] T 859ea66ebf114eaf86bc62248d768c92 P f11892e20695415d969c83d2ca192871 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 913df449ff944b7fb8e229b6a7ae1867 (127.0.98.1:35919), 77f3eb0b84f84d81873b63de3df030b5 (127.0.98.3:40601)
I20260924 21:08:19.056267 4940 tablet_bootstrap.cc:492] T 859ea66ebf114eaf86bc62248d768c92 P 913df449ff944b7fb8e229b6a7ae1867: No bootstrap required, opened a new log
I20260924 21:08:19.056325 4940 ts_tablet_manager.cc:1397] T 859ea66ebf114eaf86bc62248d768c92 P 913df449ff944b7fb8e229b6a7ae1867: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 21:08:19.057894 4940 raft_consensus.cc:348] T 859ea66ebf114eaf86bc62248d768c92 P 913df449ff944b7fb8e229b6a7ae1867 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "913df449ff944b7fb8e229b6a7ae1867" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35919 } } peers { permanent_uuid: "77f3eb0b84f84d81873b63de3df030b5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40601 } } peers { permanent_uuid: "f11892e20695415d969c83d2ca192871" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43007 } }
I20260924 21:08:19.058046 4940 raft_consensus.cc:374] T 859ea66ebf114eaf86bc62248d768c92 P 913df449ff944b7fb8e229b6a7ae1867 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 21:08:19.058075 4940 raft_consensus.cc:729] T 859ea66ebf114eaf86bc62248d768c92 P 913df449ff944b7fb8e229b6a7ae1867 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 913df449ff944b7fb8e229b6a7ae1867, State: Initialized, Role: FOLLOWER
I20260924 21:08:19.058187 4940 consensus_queue.cc:260] T 859ea66ebf114eaf86bc62248d768c92 P 913df449ff944b7fb8e229b6a7ae1867 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "913df449ff944b7fb8e229b6a7ae1867" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35919 } } peers { permanent_uuid: "77f3eb0b84f84d81873b63de3df030b5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40601 } } peers { permanent_uuid: "f11892e20695415d969c83d2ca192871" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43007 } }
I20260924 21:08:19.058424 4655 heartbeater.cc:503] Master 127.0.98.62:34109 was elected leader, sending a full tablet report...
I20260924 21:08:19.058435 4940 ts_tablet_manager.cc:1428] T 859ea66ebf114eaf86bc62248d768c92 P 913df449ff944b7fb8e229b6a7ae1867: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 21:08:19.058686 4790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:19.055659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40601 (local address 127.0.98.2:42111)
0924 21:08:19.055718 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0924 21:08:19.055728 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 21:08:19.055763 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 21:08:19.056110 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 21:08:19.056112 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 21:08:19.056113 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 21:08:19.056166 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:19.056168 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:19.056845 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:19.056846 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:19.057670 (+ 824us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:19.057672 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:19.058443 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:19.058445 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:19.058527 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:19.058530 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 21:08:19.058581 (+ 51us) client_negotiation.cc:240] Negotiation successful
0924 21:08:19.058615 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45}
I20260924 21:08:19.058883 4931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:19.055869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:42111 (local address 127.0.98.3:40601)
0924 21:08:19.056031 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0924 21:08:19.056032 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 21:08:19.056039 (+ 7us) server_negotiation.cc:407] Connection header received
0924 21:08:19.056056 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:08:19.056056 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:08:19.056063 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:08:19.056103 (+ 40us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 21:08:19.056238 (+ 135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:19.056777 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:19.057740 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:19.058335 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:19.058405 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:19.058413 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0924 21:08:19.058788 (+ 375us) server_negotiation.cc:299] Negotiation successful
0924 21:08:19.058829 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":147}
I20260924 21:08:19.059175 4875 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "859ea66ebf114eaf86bc62248d768c92" candidate_uuid: "f11892e20695415d969c83d2ca192871" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77f3eb0b84f84d81873b63de3df030b5" is_pre_election: true
I20260924 21:08:19.059305 4875 raft_consensus.cc:2454] T 859ea66ebf114eaf86bc62248d768c92 P 77f3eb0b84f84d81873b63de3df030b5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f11892e20695415d969c83d2ca192871 in term 0.
I20260924 21:08:19.059520 4675 leader_election.cc:302] T 859ea66ebf114eaf86bc62248d768c92 P f11892e20695415d969c83d2ca192871 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77f3eb0b84f84d81873b63de3df030b5, f11892e20695415d969c83d2ca192871; no voters:
I20260924 21:08:19.059615 4941 raft_consensus.cc:2781] T 859ea66ebf114eaf86bc62248d768c92 P f11892e20695415d969c83d2ca192871 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 21:08:19.059655 4941 raft_consensus.cc:482] T 859ea66ebf114eaf86bc62248d768c92 P f11892e20695415d969c83d2ca192871 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 21:08:19.059677 4941 raft_consensus.cc:3037] T 859ea66ebf114eaf86bc62248d768c92 P f11892e20695415d969c83d2ca192871 [term 0 FOLLOWER]: Advancing to term 1
I20260924 21:08:19.059782 4932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:19.055920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:53387 (local address 127.0.98.1:35919)
0924 21:08:19.055950 (+ 30us) server_negotiation.cc:206] Beginning negotiation
0924 21:08:19.055951 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 21:08:19.056286 (+ 335us) server_negotiation.cc:407] Connection header received
0924 21:08:19.056398 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:08:19.056399 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:08:19.056406 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:08:19.056428 (+ 22us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 21:08:19.056573 (+ 145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:19.057017 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:19.058919 (+ 1902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:19.059490 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:19.059525 (+ 35us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:19.059532 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0924 21:08:19.059705 (+ 173us) server_negotiation.cc:299] Negotiation successful
0924 21:08:19.059734 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13}
I20260924 21:08:19.059805 4944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:19.055659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35919 (local address 127.0.98.2:53387)
0924 21:08:19.056240 (+ 581us) negotiation.cc:107] Waiting for socket to connect
0924 21:08:19.056248 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0924 21:08:19.056295 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 21:08:19.056481 (+ 186us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 21:08:19.056482 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 21:08:19.056483 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 21:08:19.056521 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:19.056522 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:19.057735 (+ 1213us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:19.057735 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:19.058823 (+ 1088us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:19.058826 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:19.059594 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:19.059595 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:19.059654 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:19.059657 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 21:08:19.059703 (+ 46us) client_negotiation.cc:240] Negotiation successful
0924 21:08:19.059736 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":568,"mutex_wait_us":463,"thread_start_us":81,"threads_started":1}
I20260924 21:08:19.059993 4609 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "859ea66ebf114eaf86bc62248d768c92" candidate_uuid: "f11892e20695415d969c83d2ca192871" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "913df449ff944b7fb8e229b6a7ae1867" is_pre_election: true
I20260924 21:08:19.060065 4609 raft_consensus.cc:2454] T 859ea66ebf114eaf86bc62248d768c92 P 913df449ff944b7fb8e229b6a7ae1867 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f11892e20695415d969c83d2ca192871 in term 0.
I20260924 21:08:19.060297 4941 raft_consensus.cc:504] T 859ea66ebf114eaf86bc62248d768c92 P f11892e20695415d969c83d2ca192871 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "913df449ff944b7fb8e229b6a7ae1867" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35919 } } peers { permanent_uuid: "77f3eb0b84f84d81873b63de3df030b5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40601 } } peers { permanent_uuid: "f11892e20695415d969c83d2ca192871" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43007 } }
I20260924 21:08:19.060420 4941 leader_election.cc:288] T 859ea66ebf114eaf86bc62248d768c92 P f11892e20695415d969c83d2ca192871 [CANDIDATE]: Term 1 election: Requested vote from peers 913df449ff944b7fb8e229b6a7ae1867 (127.0.98.1:35919), 77f3eb0b84f84d81873b63de3df030b5 (127.0.98.3:40601)
I20260924 21:08:19.060523 4609 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "859ea66ebf114eaf86bc62248d768c92" candidate_uuid: "f11892e20695415d969c83d2ca192871" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "913df449ff944b7fb8e229b6a7ae1867"
I20260924 21:08:19.060528 4875 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "859ea66ebf114eaf86bc62248d768c92" candidate_uuid: "f11892e20695415d969c83d2ca192871" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77f3eb0b84f84d81873b63de3df030b5"
I20260924 21:08:19.060580 4609 raft_consensus.cc:3037] T 859ea66ebf114eaf86bc62248d768c92 P 913df449ff944b7fb8e229b6a7ae1867 [term 0 FOLLOWER]: Advancing to term 1
I20260924 21:08:19.060580 4875 raft_consensus.cc:3037] T 859ea66ebf114eaf86bc62248d768c92 P 77f3eb0b84f84d81873b63de3df030b5 [term 0 FOLLOWER]: Advancing to term 1
I20260924 21:08:19.061056 4609 raft_consensus.cc:2454] T 859ea66ebf114eaf86bc62248d768c92 P 913df449ff944b7fb8e229b6a7ae1867 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f11892e20695415d969c83d2ca192871 in term 1.
I20260924 21:08:19.061093 4875 raft_consensus.cc:2454] T 859ea66ebf114eaf86bc62248d768c92 P 77f3eb0b84f84d81873b63de3df030b5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f11892e20695415d969c83d2ca192871 in term 1.
I20260924 21:08:19.061223 4676 leader_election.cc:302] T 859ea66ebf114eaf86bc62248d768c92 P f11892e20695415d969c83d2ca192871 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 913df449ff944b7fb8e229b6a7ae1867, f11892e20695415d969c83d2ca192871; no voters:
I20260924 21:08:19.061307 4941 raft_consensus.cc:2781] T 859ea66ebf114eaf86bc62248d768c92 P f11892e20695415d969c83d2ca192871 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 21:08:19.061365 4941 raft_consensus.cc:686] T 859ea66ebf114eaf86bc62248d768c92 P f11892e20695415d969c83d2ca192871 [term 1 LEADER]: Becoming Leader. State: Replica: f11892e20695415d969c83d2ca192871, State: Running, Role: LEADER
I20260924 21:08:19.061429 4941 consensus_queue.cc:237] T 859ea66ebf114eaf86bc62248d768c92 P f11892e20695415d969c83d2ca192871 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "913df449ff944b7fb8e229b6a7ae1867" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35919 } } peers { permanent_uuid: "77f3eb0b84f84d81873b63de3df030b5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40601 } } peers { permanent_uuid: "f11892e20695415d969c83d2ca192871" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43007 } }
I20260924 21:08:19.062031 4464 catalog_manager.cc:5495] T 859ea66ebf114eaf86bc62248d768c92 P f11892e20695415d969c83d2ca192871 reported cstate change: term changed from 0 to 1, leader changed from <none> to f11892e20695415d969c83d2ca192871 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "f11892e20695415d969c83d2ca192871" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "913df449ff944b7fb8e229b6a7ae1867" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35919 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "77f3eb0b84f84d81873b63de3df030b5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40601 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f11892e20695415d969c83d2ca192871" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43007 } health_report { overall_health: HEALTHY } } }
I20260924 21:08:19.067564 4934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:19.065999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49808 (local address 127.0.98.2:43007)
0924 21:08:19.066026 (+ 27us) server_negotiation.cc:206] Beginning negotiation
0924 21:08:19.066027 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 21:08:19.066034 (+ 7us) server_negotiation.cc:407] Connection header received
0924 21:08:19.066082 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:08:19.066082 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:08:19.066093 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:08:19.066144 (+ 51us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 21:08:19.066211 (+ 67us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:19.066547 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:19.067092 (+ 545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:19.067227 (+ 135us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:19.067265 (+ 38us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:19.067400 (+ 135us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 21:08:19.067463 (+ 63us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 21:08:19.067508 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0924 21:08:19.067519 (+ 11us) server_negotiation.cc:299] Negotiation successful
0924 21:08:19.067530 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14}
I20260924 21:08:19.068763 4875 raft_consensus.cc:1261] T 859ea66ebf114eaf86bc62248d768c92 P 77f3eb0b84f84d81873b63de3df030b5 [term 1 FOLLOWER]: Refusing update from remote peer f11892e20695415d969c83d2ca192871: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 21:08:19.068795 4609 raft_consensus.cc:1261] T 859ea66ebf114eaf86bc62248d768c92 P 913df449ff944b7fb8e229b6a7ae1867 [term 1 FOLLOWER]: Refusing update from remote peer f11892e20695415d969c83d2ca192871: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 21:08:19.069037 4942 consensus_queue.cc:1035] T 859ea66ebf114eaf86bc62248d768c92 P f11892e20695415d969c83d2ca192871 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77f3eb0b84f84d81873b63de3df030b5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 21:08:19.069128 4941 consensus_queue.cc:1035] T 859ea66ebf114eaf86bc62248d768c92 P f11892e20695415d969c83d2ca192871 [LEADER]: Connected to new peer: Peer: permanent_uuid: "913df449ff944b7fb8e229b6a7ae1867" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35919 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 21:08:19.070796 4948 mvcc.cc:205] Tried to move back new op lower bound from 7333003669783801856 to 7333003669756067840. Current Snapshot: MvccSnapshot[applied={T|T < 7333003669783801856}]
I20260924 21:08:19.071384 4949 mvcc.cc:205] Tried to move back new op lower bound from 7333003669783801856 to 7333003669756067840. Current Snapshot: MvccSnapshot[applied={T|T < 7333003669783801856}]
I20260924 21:08:19.072465 4465 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49246:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 21:08:19.072547 4465 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49246:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 21:08:19.073083 4465 catalog_manager.cc:5782] T 00000000000000000000000000000000 P cc607ee1f27447e0a2f6eb8768cb0a2c: Sending DeleteTablet for 3 replicas of tablet 859ea66ebf114eaf86bc62248d768c92
I20260924 21:08:19.073333 4589 tablet_service.cc:1514] Processing DeleteTablet for tablet 859ea66ebf114eaf86bc62248d768c92 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 21:08:19 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:41664
I20260924 21:08:19.073338 4722 tablet_service.cc:1514] Processing DeleteTablet for tablet 859ea66ebf114eaf86bc62248d768c92 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 21:08:19 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:49800
I20260924 21:08:19.073513 4959 tablet_replica.cc:330] T 859ea66ebf114eaf86bc62248d768c92 P 913df449ff944b7fb8e229b6a7ae1867: stopping tablet replica
I20260924 21:08:19.073524 4855 tablet_service.cc:1514] Processing DeleteTablet for tablet 859ea66ebf114eaf86bc62248d768c92 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 21:08:19 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:47640
I20260924 21:08:19.073566 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAzNNjS/build/release/bin/kudu with pid 4527
I20260924 21:08:19.073598 4959 raft_consensus.cc:2229] T 859ea66ebf114eaf86bc62248d768c92 P 913df449ff944b7fb8e229b6a7ae1867 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 21:08:19.077883 4958 tablet_replica.cc:330] T 859ea66ebf114eaf86bc62248d768c92 P f11892e20695415d969c83d2ca192871: stopping tablet replica
I20260924 21:08:19.077967 4958 raft_consensus.cc:2229] T 859ea66ebf114eaf86bc62248d768c92 P f11892e20695415d969c83d2ca192871 [term 1 LEADER]: Raft consensus shutting down.
I20260924 21:08:19.078104 4958 raft_consensus.cc:2258] T 859ea66ebf114eaf86bc62248d768c92 P f11892e20695415d969c83d2ca192871 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 21:08:19.078436 4958 ts_tablet_manager.cc:1905] T 859ea66ebf114eaf86bc62248d768c92 P f11892e20695415d969c83d2ca192871: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260924 21:08:19.079171 4450 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:35919 (error 108)
W20260924 21:08:19.079628 4930 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 21:08:19.079469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35919 (local address 127.0.0.1:41666)
0924 21:08:19.079512 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0924 21:08:19.079521 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 21:08:19.079558 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 21:08:19.079594 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:35919: BlockingWrite error: write error: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":29}
W20260924 21:08:19.079804 4450 catalog_manager.cc:4584] TS 913df449ff944b7fb8e229b6a7ae1867 (127.0.98.1:35919): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 859ea66ebf114eaf86bc62248d768c92: Network error: Client connection negotiation failed: client connection to 127.0.98.1:35919: BlockingWrite error: write error: Connection reset by peer (error 104)
I20260924 21:08:19.079816 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAzNNjS/build/release/bin/kudu with pid 4660
I20260924 21:08:19.080250 4960 tablet_replica.cc:330] T 859ea66ebf114eaf86bc62248d768c92 P 77f3eb0b84f84d81873b63de3df030b5: stopping tablet replica
I20260924 21:08:19.080324 4960 raft_consensus.cc:2229] T 859ea66ebf114eaf86bc62248d768c92 P 77f3eb0b84f84d81873b63de3df030b5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 21:08:19.080406 4960 raft_consensus.cc:2258] T 859ea66ebf114eaf86bc62248d768c92 P 77f3eb0b84f84d81873b63de3df030b5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 21:08:19.081163 4960 ts_tablet_manager.cc:1905] T 859ea66ebf114eaf86bc62248d768c92 P 77f3eb0b84f84d81873b63de3df030b5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 21:08:19.082334 4960 ts_tablet_manager.cc:1918] T 859ea66ebf114eaf86bc62248d768c92 P 77f3eb0b84f84d81873b63de3df030b5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 21:08:19.082412 4960 log.cc:1196] T 859ea66ebf114eaf86bc62248d768c92 P 77f3eb0b84f84d81873b63de3df030b5: Deleting WAL directory at /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790284076800039-392-0/minicluster-data/ts-2/wal/wals/859ea66ebf114eaf86bc62248d768c92
I20260924 21:08:19.082633 4960 ts_tablet_manager.cc:1939] T 859ea66ebf114eaf86bc62248d768c92 P 77f3eb0b84f84d81873b63de3df030b5: Deleting consensus metadata
I20260924 21:08:19.083000 4449 catalog_manager.cc:4841] TS 77f3eb0b84f84d81873b63de3df030b5 (127.0.98.3:40601): tablet 859ea66ebf114eaf86bc62248d768c92 (table test-table [id=2bd494b552fe4a5e9191a9b53c23bb0a]) successfully deleted
I20260924 21:08:19.084829 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAzNNjS/build/release/bin/kudu with pid 4793
I20260924 21:08:19.084834 4930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:19.084743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43007 (local address 127.0.0.1:49824)
0924 21:08:19.084782 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0924 21:08:19.084814 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:43007: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
I20260924 21:08:19.088943 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAzNNjS/build/release/bin/kudu with pid 4434
2026-09-24T21:08:19Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (1625 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-24T21:08:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T21:08:19Z Disabled control of system clock
I20260924 21:08:19.113595 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAzNNjS/build/release/bin/kudu
/tmp/dist-test-taskAzNNjS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790284076800039-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790284076800039-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790284076800039-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790284076800039-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37751
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:38461
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37751
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260924 21:08:19.182826 4965 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 21:08:19.182984 4965 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 21:08:19.183003 4965 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 21:08:19.183917 4965 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 21:08:19.183946 4965 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 21:08:19.183957 4965 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 21:08:19.183969 4965 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 21:08:19.183979 4965 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 21:08:19.185007 4965 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38461
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790284076800039-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790284076800039-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37751
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37751
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790284076800039-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4965
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790284076800039-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 19:49:06 UTC on ccycloud.quasar-vhqjoc.root.comops.site
build id 4670864
I20260924 21:08:19.185151 4965 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 21:08:19.185324 4965 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 21:08:19.189414 4970 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 21:08:19.189410 4971 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 21:08:19.189440 4965 server_base.cc:1034] running on GCE node
W20260924 21:08:19.189383 4973 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 21:08:19.189724 4965 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 21:08:19.190011 4965 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 21:08:19.191156 4965 hybrid_clock.cc:648] HybridClock initialized: now 1790284099191146 us; error 25 us; skew 500 ppm
I20260924 21:08:19.192203 4965 webserver.cc:460] Webserver started at http://127.0.98.62:44989/ using document root <none> and password file <none>
I20260924 21:08:19.192406 4965 fs_manager.cc:359] Metadata directory not provided
I20260924 21:08:19.192443 4965 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 21:08:19.192562 4965 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 21:08:19.193388 4965 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790284076800039-392-0/minicluster-data/master-0/data/instance:
uuid: "924f420cede648c8b0e3311467393b0c"
format_stamp: "Formatted at 2026-09-24 21:08:19 on dist-test-slave-c7tk"
server_key: "41515957dff0fa42e1f87dd2af31f0f2"
server_key_iv: "5b068e27b1a24bac8f07b679850f7f34"
server_key_version: "encryptionkey@0"
I20260924 21:08:19.193717 4965 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790284076800039-392-0/minicluster-data/master-0/wal/instance:
uuid: "924f420cede648c8b0e3311467393b0c"
format_stamp: "Formatted at 2026-09-24 21:08:19 on dist-test-slave-c7tk"
server_key: "41515957dff0fa42e1f87dd2af31f0f2"
server_key_iv: "5b068e27b1a24bac8f07b679850f7f34"
server_key_version: "encryptionkey@0"
I20260924 21:08:19.194880 4965 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 21:08:19.195447 4979 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 21:08:19.195585 4965 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 21:08:19.195662 4965 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790284076800039-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790284076800039-392-0/minicluster-data/master-0/wal
uuid: "924f420cede648c8b0e3311467393b0c"
format_stamp: "Formatted at 2026-09-24 21:08:19 on dist-test-slave-c7tk"
server_key: "41515957dff0fa42e1f87dd2af31f0f2"
server_key_iv: "5b068e27b1a24bac8f07b679850f7f34"
server_key_version: "encryptionkey@0"
I20260924 21:08:19.195727 4965 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790284076800039-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790284076800039-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790284076800039-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 21:08:19.198526 4965 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 21:08:19.198750 4965 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 21:08:19.198885 4965 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 21:08:19.201954 4965 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37751
I20260924 21:08:19.202005 5031 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37751 every 8 connection(s)
I20260924 21:08:19.202256 4965 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790284076800039-392-0/minicluster-data/master-0/data/info.pb
I20260924 21:08:19.202757 5032 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 21:08:19.204800 5032 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 924f420cede648c8b0e3311467393b0c: Bootstrap starting.
I20260924 21:08:19.205283 5032 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 924f420cede648c8b0e3311467393b0c: Neither blocks nor log segments found. Creating new log.
I20260924 21:08:19.205542 5032 log.cc:824] T 00000000000000000000000000000000 P 924f420cede648c8b0e3311467393b0c: Log is configured to *not* fsync() on all Append() calls
I20260924 21:08:19.206096 5032 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 924f420cede648c8b0e3311467393b0c: No bootstrap required, opened a new log
I20260924 21:08:19.206766 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAzNNjS/build/release/bin/kudu as pid 4965
I20260924 21:08:19.206857 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790284076800039-392-0/minicluster-data/master-0/wal/instance
I20260924 21:08:19.207011 392 external_mini_cluster.cc:1428] Setting key 6b7b737df5dad068cbd257f8851bdad8
I20260924 21:08:19.207852 5032 raft_consensus.cc:348] T 00000000000000000000000000000000 P 924f420cede648c8b0e3311467393b0c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "924f420cede648c8b0e3311467393b0c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37751 } }
I20260924 21:08:19.207994 5032 raft_consensus.cc:374] T 00000000000000000000000000000000 P 924f420cede648c8b0e3311467393b0c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 21:08:19.208055 5032 raft_consensus.cc:729] T 00000000000000000000000000000000 P 924f420cede648c8b0e3311467393b0c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 924f420cede648c8b0e3311467393b0c, State: Initialized, Role: FOLLOWER
I20260924 21:08:19.208169 5032 consensus_queue.cc:260] T 00000000000000000000000000000000 P 924f420cede648c8b0e3311467393b0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "924f420cede648c8b0e3311467393b0c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37751 } }
I20260924 21:08:19.208252 5032 raft_consensus.cc:388] T 00000000000000000000000000000000 P 924f420cede648c8b0e3311467393b0c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 21:08:19.208288 5032 raft_consensus.cc:482] T 00000000000000000000000000000000 P 924f420cede648c8b0e3311467393b0c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 21:08:19.208333 5032 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 924f420cede648c8b0e3311467393b0c [term 0 FOLLOWER]: Advancing to term 1
I20260924 21:08:19.208972 5032 raft_consensus.cc:504] T 00000000000000000000000000000000 P 924f420cede648c8b0e3311467393b0c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "924f420cede648c8b0e3311467393b0c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37751 } }
I20260924 21:08:19.209055 5036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:19.207437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49894 (local address 127.0.98.62:37751)
0924 21:08:19.207621 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0924 21:08:19.207628 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 21:08:19.207657 (+ 29us) server_negotiation.cc:407] Connection header received
0924 21:08:19.207817 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:08:19.207822 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:08:19.207871 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:08:19.208004 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:08:19.208632 (+ 628us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 21:08:19.208635 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 21:08:19.208635 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 21:08:19.208640 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 21:08:19.208679 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 21:08:19.208693 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 21:08:19.208767 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0924 21:08:19.208804 (+ 37us) server_negotiation.cc:299] Negotiation successful
0924 21:08:19.208853 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112,"thread_start_us":73,"threads_started":1}
I20260924 21:08:19.209070 5032 leader_election.cc:302] T 00000000000000000000000000000000 P 924f420cede648c8b0e3311467393b0c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 924f420cede648c8b0e3311467393b0c; no voters:
I20260924 21:08:19.209241 5032 leader_election.cc:288] T 00000000000000000000000000000000 P 924f420cede648c8b0e3311467393b0c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 21:08:19.209436 5032 sys_catalog.cc:564] T 00000000000000000000000000000000 P 924f420cede648c8b0e3311467393b0c [sys.catalog]: configured and running, proceeding with master startup.
I20260924 21:08:19.209523 5037 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 924f420cede648c8b0e3311467393b0c [term 1 FOLLOWER]: Leader election won for term 1
I20260924 21:08:19.209700 5037 raft_consensus.cc:686] T 00000000000000000000000000000000 P 924f420cede648c8b0e3311467393b0c [term 1 LEADER]: Becoming Leader. State: Replica: 924f420cede648c8b0e3311467393b0c, State: Running, Role: LEADER
I20260924 21:08:19.209789 5037 consensus_queue.cc:237] T 00000000000000000000000000000000 P 924f420cede648c8b0e3311467393b0c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "924f420cede648c8b0e3311467393b0c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37751 } }
I20260924 21:08:19.210131 5038 sys_catalog.cc:455] T 00000000000000000000000000000000 P 924f420cede648c8b0e3311467393b0c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "924f420cede648c8b0e3311467393b0c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "924f420cede648c8b0e3311467393b0c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37751 } } }
I20260924 21:08:19.210215 5038 sys_catalog.cc:458] T 00000000000000000000000000000000 P 924f420cede648c8b0e3311467393b0c [sys.catalog]: This master's current role is: LEADER
I20260924 21:08:19.210139 5039 sys_catalog.cc:455] T 00000000000000000000000000000000 P 924f420cede648c8b0e3311467393b0c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 924f420cede648c8b0e3311467393b0c. Latest consensus state: current_term: 1 leader_uuid: "924f420cede648c8b0e3311467393b0c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "924f420cede648c8b0e3311467393b0c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37751 } } }
I20260924 21:08:19.210341 5039 sys_catalog.cc:458] T 00000000000000000000000000000000 P 924f420cede648c8b0e3311467393b0c [sys.catalog]: This master's current role is: LEADER
I20260924 21:08:19.210449 5047 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 21:08:19.210934 5047 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 21:08:19.212124 5047 catalog_manager.cc:1347] Generated new cluster ID: b259edb15bdb438ca78152d8179b1296
I20260924 21:08:19.212169 5047 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 21:08:19.225773 5047 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 21:08:19.225921 5047 catalog_manager.cc:1504] Loading token signing keys...
I20260924 21:08:19.229256 5047 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 924f420cede648c8b0e3311467393b0c: Generated new TSK 0
I20260924 21:08:19.229413 5047 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 21:08:19.232023 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAzNNjS/build/release/bin/kudu
/tmp/dist-test-taskAzNNjS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790284076800039-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790284076800039-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790284076800039-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790284076800039-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37751
--builtin_ntp_servers=127.0.98.20:38461
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260924 21:08:19.300836 5056 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 21:08:19.300999 5056 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 21:08:19.301017 5056 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 21:08:19.302027 5056 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 21:08:19.302058 5056 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 21:08:19.302080 5056 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 21:08:19.303169 5056 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38461
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790284076800039-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790284076800039-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790284076800039-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37751
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5056
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790284076800039-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 19:49:06 UTC on ccycloud.quasar-vhqjoc.root.comops.site
build id 4670864
I20260924 21:08:19.303355 5056 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 21:08:19.303591 5056 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 21:08:19.307634 5062 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 21:08:19.307636 5061 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 21:08:19.307796 5064 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 21:08:19.307934 5056 server_base.cc:1034] running on GCE node
I20260924 21:08:19.308204 5056 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 21:08:19.308434 5056 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 21:08:19.309579 5056 hybrid_clock.cc:648] HybridClock initialized: now 1790284099309569 us; error 27 us; skew 500 ppm
I20260924 21:08:19.310760 5056 webserver.cc:460] Webserver started at http://127.0.98.1:46099/ using document root <none> and password file <none>
I20260924 21:08:19.310982 5056 fs_manager.cc:359] Metadata directory not provided
I20260924 21:08:19.311031 5056 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 21:08:19.311138 5056 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 21:08:19.312220 5056 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790284076800039-392-0/minicluster-data/ts-0/data/instance:
uuid: "9ec5861005e44c0ab8fca59defefb89b"
format_stamp: "Formatted at 2026-09-24 21:08:19 on dist-test-slave-c7tk"
server_key: "34f0ce539d5c72e2fdd722b4401375ae"
server_key_iv: "0e0b74bf0db805e59345952fd4b92c61"
server_key_version: "encryptionkey@0"
I20260924 21:08:19.312584 5056 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790284076800039-392-0/minicluster-data/ts-0/wal/instance:
uuid: "9ec5861005e44c0ab8fca59defefb89b"
format_stamp: "Formatted at 2026-09-24 21:08:19 on dist-test-slave-c7tk"
server_key: "34f0ce539d5c72e2fdd722b4401375ae"
server_key_iv: "0e0b74bf0db805e59345952fd4b92c61"
server_key_version: "encryptionkey@0"
I20260924 21:08:19.315136 5056 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20260924 21:08:19.316455 5070 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 21:08:19.316604 5056 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 21:08:19.316681 5056 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790284076800039-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790284076800039-392-0/minicluster-data/ts-0/wal
uuid: "9ec5861005e44c0ab8fca59defefb89b"
format_stamp: "Formatted at 2026-09-24 21:08:19 on dist-test-slave-c7tk"
server_key: "34f0ce539d5c72e2fdd722b4401375ae"
server_key_iv: "0e0b74bf0db805e59345952fd4b92c61"
server_key_version: "encryptionkey@0"
I20260924 21:08:19.316740 5056 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790284076800039-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790284076800039-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790284076800039-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 21:08:19.319703 5056 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 21:08:19.319953 5056 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 21:08:19.320094 5056 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 21:08:19.320309 5056 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 21:08:19.320616 5056 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 21:08:19.320660 5056 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 21:08:19.320695 5056 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 21:08:19.320719 5056 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 21:08:19.325896 5056 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:34621
I20260924 21:08:19.325949 5183 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:34621 every 8 connection(s)
I20260924 21:08:19.326236 5056 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790284076800039-392-0/minicluster-data/ts-0/data/info.pb
I20260924 21:08:19.328670 5036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:19.326828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:33585 (local address 127.0.98.62:37751)
0924 21:08:19.326883 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0924 21:08:19.326884 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 21:08:19.327074 (+ 190us) server_negotiation.cc:407] Connection header received
0924 21:08:19.327458 (+ 384us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:08:19.327459 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:08:19.327464 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:08:19.327502 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:08:19.328515 (+ 1013us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 21:08:19.328515 (+ 0us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 21:08:19.328516 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 21:08:19.328518 (+ 2us) server_negotiation.cc:922] calling sasl_server_start()
0924 21:08:19.328529 (+ 11us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 21:08:19.328532 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 21:08:19.328571 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0924 21:08:19.328614 (+ 43us) server_negotiation.cc:299] Negotiation successful
0924 21:08:19.328625 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260924 21:08:19.328840 5186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:19.326826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37751 (local address 127.0.98.1:33585)
0924 21:08:19.326962 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0924 21:08:19.326986 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0924 21:08:19.327376 (+ 390us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 21:08:19.327544 (+ 168us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 21:08:19.327547 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 21:08:19.327553 (+ 6us) client_negotiation.cc:189] Negotiated authn=SASL
0924 21:08:19.328401 (+ 848us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 21:08:19.328402 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0924 21:08:19.328426 (+ 24us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 21:08:19.328451 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 21:08:19.328570 (+ 119us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 21:08:19.328572 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 21:08:19.328574 (+ 2us) client_negotiation.cc:769] Sending connection context
0924 21:08:19.328621 (+ 47us) client_negotiation.cc:240] Negotiation successful
0924 21:08:19.328662 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":91,"thread_start_us":72,"threads_started":1}
I20260924 21:08:19.329306 5184 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37751
I20260924 21:08:19.329378 5184 heartbeater.cc:464] Registering TS with master...
I20260924 21:08:19.329468 5184 heartbeater.cc:511] Master 127.0.98.62:37751 requested a full tablet report, sending...
I20260924 21:08:19.329859 4996 ts_manager.cc:194] Registered new tserver with Master: 9ec5861005e44c0ab8fca59defefb89b (127.0.98.1:34621)
I20260924 21:08:19.335868 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAzNNjS/build/release/bin/kudu as pid 5056
I20260924 21:08:19.335939 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790284076800039-392-0/minicluster-data/ts-0/wal/instance
I20260924 21:08:19.336051 392 external_mini_cluster.cc:1428] Setting key 1edae479b77658c8d7fd089e6a395f84
I20260924 21:08:19.336710 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAzNNjS/build/release/bin/kudu
/tmp/dist-test-taskAzNNjS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790284076800039-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790284076800039-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790284076800039-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790284076800039-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37751
--builtin_ntp_servers=127.0.98.20:38461
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260924 21:08:19.407498 5187 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 21:08:19.407670 5187 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 21:08:19.407691 5187 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 21:08:19.408845 5187 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 21:08:19.408887 5187 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 21:08:19.408941 5187 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 21:08:19.410329 5187 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38461
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790284076800039-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790284076800039-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790284076800039-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37751
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5187
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790284076800039-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 19:49:06 UTC on ccycloud.quasar-vhqjoc.root.comops.site
build id 4670864
I20260924 21:08:19.410529 5187 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 21:08:19.410772 5187 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 21:08:19.415127 5187 server_base.cc:1034] running on GCE node
W20260924 21:08:19.415154 5195 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 21:08:19.415175 5193 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 21:08:19.415308 5192 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 21:08:19.415488 5187 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 21:08:19.415691 5187 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 21:08:19.416831 5187 hybrid_clock.cc:648] HybridClock initialized: now 1790284099416825 us; error 30 us; skew 500 ppm
I20260924 21:08:19.417881 5187 webserver.cc:460] Webserver started at http://127.0.98.2:34657/ using document root <none> and password file <none>
I20260924 21:08:19.418080 5187 fs_manager.cc:359] Metadata directory not provided
I20260924 21:08:19.418160 5187 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 21:08:19.418283 5187 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 21:08:19.419294 5187 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790284076800039-392-0/minicluster-data/ts-1/data/instance:
uuid: "6720c15219e04124b3ebc09c050bfa24"
format_stamp: "Formatted at 2026-09-24 21:08:19 on dist-test-slave-c7tk"
server_key: "128548eeab1ffdf694380a11751c75da"
server_key_iv: "87f2ab35ff80769fe61aee89c8ab9b69"
server_key_version: "encryptionkey@0"
I20260924 21:08:19.419610 5187 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790284076800039-392-0/minicluster-data/ts-1/wal/instance:
uuid: "6720c15219e04124b3ebc09c050bfa24"
format_stamp: "Formatted at 2026-09-24 21:08:19 on dist-test-slave-c7tk"
server_key: "128548eeab1ffdf694380a11751c75da"
server_key_iv: "87f2ab35ff80769fe61aee89c8ab9b69"
server_key_version: "encryptionkey@0"
I20260924 21:08:19.420783 5187 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 21:08:19.421372 5201 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 21:08:19.421512 5187 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 21:08:19.421593 5187 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790284076800039-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790284076800039-392-0/minicluster-data/ts-1/wal
uuid: "6720c15219e04124b3ebc09c050bfa24"
format_stamp: "Formatted at 2026-09-24 21:08:19 on dist-test-slave-c7tk"
server_key: "128548eeab1ffdf694380a11751c75da"
server_key_iv: "87f2ab35ff80769fe61aee89c8ab9b69"
server_key_version: "encryptionkey@0"
I20260924 21:08:19.421662 5187 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790284076800039-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790284076800039-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790284076800039-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 21:08:19.424393 5187 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 21:08:19.424597 5187 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 21:08:19.424707 5187 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 21:08:19.424878 5187 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 21:08:19.425129 5187 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 21:08:19.425181 5187 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 21:08:19.425213 5187 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 21:08:19.425239 5187 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 21:08:19.430244 5187 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:43977
I20260924 21:08:19.430282 5314 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:43977 every 8 connection(s)
I20260924 21:08:19.430531 5187 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790284076800039-392-0/minicluster-data/ts-1/data/info.pb
I20260924 21:08:19.432624 5036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:19.431107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:41655 (local address 127.0.98.62:37751)
0924 21:08:19.431159 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0924 21:08:19.431161 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 21:08:19.431325 (+ 164us) server_negotiation.cc:407] Connection header received
0924 21:08:19.431639 (+ 314us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:08:19.431640 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:08:19.431644 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:08:19.431685 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:08:19.432470 (+ 785us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 21:08:19.432472 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 21:08:19.432472 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 21:08:19.432475 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0924 21:08:19.432486 (+ 11us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 21:08:19.432489 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 21:08:19.432532 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0924 21:08:19.432572 (+ 40us) server_negotiation.cc:299] Negotiation successful
0924 21:08:19.432582 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260924 21:08:19.432785 5317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:19.431091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37751 (local address 127.0.98.2:41655)
0924 21:08:19.431230 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0924 21:08:19.431259 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0924 21:08:19.431576 (+ 317us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 21:08:19.431729 (+ 153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 21:08:19.431731 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 21:08:19.431737 (+ 6us) client_negotiation.cc:189] Negotiated authn=SASL
0924 21:08:19.432369 (+ 632us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 21:08:19.432369 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0924 21:08:19.432389 (+ 20us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 21:08:19.432411 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 21:08:19.432532 (+ 121us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 21:08:19.432534 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 21:08:19.432536 (+ 2us) client_negotiation.cc:769] Sending connection context
0924 21:08:19.432576 (+ 40us) client_negotiation.cc:240] Negotiation successful
0924 21:08:19.432611 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":89,"thread_start_us":69,"threads_started":1}
I20260924 21:08:19.433169 5315 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37751
I20260924 21:08:19.433243 5315 heartbeater.cc:464] Registering TS with master...
I20260924 21:08:19.433353 5315 heartbeater.cc:511] Master 127.0.98.62:37751 requested a full tablet report, sending...
I20260924 21:08:19.433638 4996 ts_manager.cc:194] Registered new tserver with Master: 6720c15219e04124b3ebc09c050bfa24 (127.0.98.2:43977)
I20260924 21:08:19.440634 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAzNNjS/build/release/bin/kudu as pid 5187
I20260924 21:08:19.440706 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790284076800039-392-0/minicluster-data/ts-1/wal/instance
I20260924 21:08:19.440852 392 external_mini_cluster.cc:1428] Setting key 38af62c48135d7dcbe12203b5f365ff0
I20260924 21:08:19.441605 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAzNNjS/build/release/bin/kudu
/tmp/dist-test-taskAzNNjS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790284076800039-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790284076800039-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790284076800039-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790284076800039-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37751
--builtin_ntp_servers=127.0.98.20:38461
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260924 21:08:19.509451 5318 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 21:08:19.509613 5318 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 21:08:19.509642 5318 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 21:08:19.510661 5318 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 21:08:19.510705 5318 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 21:08:19.510741 5318 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 21:08:19.511861 5318 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38461
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790284076800039-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790284076800039-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790284076800039-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37751
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5318
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790284076800039-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 19:49:06 UTC on ccycloud.quasar-vhqjoc.root.comops.site
build id 4670864
I20260924 21:08:19.512079 5318 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 21:08:19.512295 5318 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 21:08:19.516361 5324 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 21:08:19.516357 5323 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 21:08:19.516489 5318 server_base.cc:1034] running on GCE node
W20260924 21:08:19.516359 5326 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 21:08:19.516654 5318 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 21:08:19.516844 5318 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 21:08:19.517982 5318 hybrid_clock.cc:648] HybridClock initialized: now 1790284099517970 us; error 22 us; skew 500 ppm
I20260924 21:08:19.519090 5318 webserver.cc:460] Webserver started at http://127.0.98.3:32825/ using document root <none> and password file <none>
I20260924 21:08:19.519320 5318 fs_manager.cc:359] Metadata directory not provided
I20260924 21:08:19.519373 5318 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 21:08:19.519479 5318 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 21:08:19.520328 5318 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790284076800039-392-0/minicluster-data/ts-2/data/instance:
uuid: "8952ed545f774f86b20c86e7f5d26218"
format_stamp: "Formatted at 2026-09-24 21:08:19 on dist-test-slave-c7tk"
server_key: "9cd5036ea15263b8ae4c4bb824cae262"
server_key_iv: "ed9190dfe34b5ff84dfcbff2bb8ea904"
server_key_version: "encryptionkey@0"
I20260924 21:08:19.520615 5318 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790284076800039-392-0/minicluster-data/ts-2/wal/instance:
uuid: "8952ed545f774f86b20c86e7f5d26218"
format_stamp: "Formatted at 2026-09-24 21:08:19 on dist-test-slave-c7tk"
server_key: "9cd5036ea15263b8ae4c4bb824cae262"
server_key_iv: "ed9190dfe34b5ff84dfcbff2bb8ea904"
server_key_version: "encryptionkey@0"
I20260924 21:08:19.521648 5318 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 21:08:19.522292 5332 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 21:08:19.522437 5318 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 21:08:19.522511 5318 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790284076800039-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790284076800039-392-0/minicluster-data/ts-2/wal
uuid: "8952ed545f774f86b20c86e7f5d26218"
format_stamp: "Formatted at 2026-09-24 21:08:19 on dist-test-slave-c7tk"
server_key: "9cd5036ea15263b8ae4c4bb824cae262"
server_key_iv: "ed9190dfe34b5ff84dfcbff2bb8ea904"
server_key_version: "encryptionkey@0"
I20260924 21:08:19.522578 5318 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790284076800039-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790284076800039-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790284076800039-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 21:08:19.525741 5318 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 21:08:19.525969 5318 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 21:08:19.526085 5318 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 21:08:19.526288 5318 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 21:08:19.526547 5318 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 21:08:19.526595 5318 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 21:08:19.526631 5318 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 21:08:19.526659 5318 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 21:08:19.531842 5318 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:36319
I20260924 21:08:19.531872 5445 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:36319 every 8 connection(s)
I20260924 21:08:19.532122 5318 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790284076800039-392-0/minicluster-data/ts-2/data/info.pb
I20260924 21:08:19.534515 5036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:19.532661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:50593 (local address 127.0.98.62:37751)
0924 21:08:19.532711 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0924 21:08:19.532713 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 21:08:19.532909 (+ 196us) server_negotiation.cc:407] Connection header received
0924 21:08:19.533271 (+ 362us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:08:19.533273 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:08:19.533279 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:08:19.533332 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:08:19.534281 (+ 949us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 21:08:19.534282 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 21:08:19.534283 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 21:08:19.534287 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 21:08:19.534301 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 21:08:19.534305 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 21:08:19.534359 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0924 21:08:19.534437 (+ 78us) server_negotiation.cc:299] Negotiation successful
0924 21:08:19.534452 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260924 21:08:19.534665 5448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:19.532664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37751 (local address 127.0.98.3:50593)
0924 21:08:19.532812 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0924 21:08:19.532836 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0924 21:08:19.533189 (+ 353us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 21:08:19.533385 (+ 196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 21:08:19.533388 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 21:08:19.533396 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0924 21:08:19.534158 (+ 762us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 21:08:19.534158 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0924 21:08:19.534180 (+ 22us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 21:08:19.534202 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 21:08:19.534357 (+ 155us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 21:08:19.534361 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 21:08:19.534364 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 21:08:19.534446 (+ 82us) client_negotiation.cc:240] Negotiation successful
0924 21:08:19.534495 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":78,"threads_started":1}
I20260924 21:08:19.535027 5446 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37751
I20260924 21:08:19.535115 5446 heartbeater.cc:464] Registering TS with master...
I20260924 21:08:19.535142 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAzNNjS/build/release/bin/kudu as pid 5318
I20260924 21:08:19.535234 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790284076800039-392-0/minicluster-data/ts-2/wal/instance
I20260924 21:08:19.535254 5446 heartbeater.cc:511] Master 127.0.98.62:37751 requested a full tablet report, sending...
I20260924 21:08:19.535406 392 external_mini_cluster.cc:1428] Setting key b6ff29448b784992846661920ee0c848
I20260924 21:08:19.535578 4996 ts_manager.cc:194] Registered new tserver with Master: 8952ed545f774f86b20c86e7f5d26218 (127.0.98.3:36319)
I20260924 21:08:19.536087 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 21:08:19.538026 5036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:19.537174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49908 (local address 127.0.98.62:37751)
0924 21:08:19.537237 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0924 21:08:19.537238 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 21:08:19.537300 (+ 62us) server_negotiation.cc:407] Connection header received
0924 21:08:19.537342 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:08:19.537342 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:08:19.537345 (+ 3us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:08:19.537383 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:08:19.537858 (+ 475us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 21:08:19.537859 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 21:08:19.537859 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 21:08:19.537861 (+ 2us) server_negotiation.cc:922] calling sasl_server_start()
0924 21:08:19.537871 (+ 10us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 21:08:19.537874 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 21:08:19.537915 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0924 21:08:19.537945 (+ 30us) server_negotiation.cc:299] Negotiation successful
0924 21:08:19.537953 (+ 8us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260924 21:08:19.539600 4995 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49908:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 21:08:19.540916 4995 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 21:08:19.543969 5454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:19.542652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43977 (local address 127.0.0.1:37124)
0924 21:08:19.542855 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0924 21:08:19.542865 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 21:08:19.542903 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 21:08:19.543169 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 21:08:19.543174 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 21:08:19.543178 (+ 4us) client_negotiation.cc:189] Negotiated authn=SASL
0924 21:08:19.543586 (+ 408us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 21:08:19.543586 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0924 21:08:19.543598 (+ 12us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 21:08:19.543616 (+ 18us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 21:08:19.543835 (+ 219us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 21:08:19.543840 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 21:08:19.543842 (+ 2us) client_negotiation.cc:769] Sending connection context
0924 21:08:19.543891 (+ 49us) client_negotiation.cc:240] Negotiation successful
0924 21:08:19.543901 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":71,"threads_started":1}
I20260924 21:08:19.544094 5458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:19.542831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34020 (local address 127.0.98.3:36319)
0924 21:08:19.542941 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0924 21:08:19.542944 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 21:08:19.543110 (+ 166us) server_negotiation.cc:407] Connection header received
0924 21:08:19.543127 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:08:19.543128 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:08:19.543134 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:08:19.543191 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:08:19.543797 (+ 606us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 21:08:19.543802 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 21:08:19.543802 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 21:08:19.543806 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 21:08:19.543834 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 21:08:19.543851 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 21:08:19.543895 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0924 21:08:19.543997 (+ 102us) server_negotiation.cc:299] Negotiation successful
0924 21:08:19.544008 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85,"thread_start_us":67,"threads_started":1}
I20260924 21:08:19.544167 5459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:19.542904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37124 (local address 127.0.98.2:43977)
0924 21:08:19.543040 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0924 21:08:19.543042 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 21:08:19.543049 (+ 7us) server_negotiation.cc:407] Connection header received
0924 21:08:19.543068 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:08:19.543069 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:08:19.543075 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:08:19.543159 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:08:19.543692 (+ 533us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 21:08:19.543698 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 21:08:19.543698 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 21:08:19.543704 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 21:08:19.543735 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 21:08:19.543761 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 21:08:19.543812 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0924 21:08:19.544068 (+ 256us) server_negotiation.cc:299] Negotiation successful
0924 21:08:19.544085 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112,"thread_start_us":79,"threads_started":1}
I20260924 21:08:19.544229 5457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:19.542792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36319 (local address 127.0.0.1:34020)
0924 21:08:19.543050 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0924 21:08:19.543057 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0924 21:08:19.543092 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 21:08:19.543417 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 21:08:19.543417 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 21:08:19.543420 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0924 21:08:19.543713 (+ 293us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 21:08:19.543713 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0924 21:08:19.543719 (+ 6us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 21:08:19.543732 (+ 13us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 21:08:19.543917 (+ 185us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 21:08:19.543917 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 21:08:19.543918 (+ 1us) client_negotiation.cc:769] Sending connection context
0924 21:08:19.543956 (+ 38us) client_negotiation.cc:240] Negotiation successful
0924 21:08:19.543968 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":46,"threads_started":1}
I20260924 21:08:19.544662 5455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:19.542639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34621 (local address 127.0.0.1:48274)
0924 21:08:19.542766 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0924 21:08:19.542796 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0924 21:08:19.542841 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 21:08:19.543712 (+ 871us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 21:08:19.543713 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 21:08:19.543715 (+ 2us) client_negotiation.cc:189] Negotiated authn=SASL
0924 21:08:19.544103 (+ 388us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 21:08:19.544104 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0924 21:08:19.544108 (+ 4us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 21:08:19.544117 (+ 9us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 21:08:19.544553 (+ 436us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 21:08:19.544554 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 21:08:19.544554 (+ 0us) client_negotiation.cc:769] Sending connection context
0924 21:08:19.544604 (+ 50us) client_negotiation.cc:240] Negotiation successful
0924 21:08:19.544614 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":40,"threads_started":1}
I20260924 21:08:19.544693 5456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:19.542706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48274 (local address 127.0.98.1:34621)
0924 21:08:19.543194 (+ 488us) server_negotiation.cc:206] Beginning negotiation
0924 21:08:19.543197 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 21:08:19.543206 (+ 9us) server_negotiation.cc:407] Connection header received
0924 21:08:19.543229 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:08:19.543230 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:08:19.543238 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:08:19.543854 (+ 616us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:08:19.544453 (+ 599us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 21:08:19.544456 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 21:08:19.544456 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 21:08:19.544459 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0924 21:08:19.544481 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 21:08:19.544491 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 21:08:19.544548 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0924 21:08:19.544609 (+ 61us) server_negotiation.cc:299] Negotiation successful
0924 21:08:19.544621 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":453,"thread_start_us":72,"threads_started":1}
I20260924 21:08:19.544688 5249 tablet_service.cc:1467] Processing CreateTablet for tablet 0d9ffedd676743aaa142fcd78650a998 (DEFAULT_TABLE table=test-table [id=f440004f822d484894c3104918b34bdc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 21:08:19.544751 5380 tablet_service.cc:1467] Processing CreateTablet for tablet 0d9ffedd676743aaa142fcd78650a998 (DEFAULT_TABLE table=test-table [id=f440004f822d484894c3104918b34bdc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 21:08:19.544952 5249 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0d9ffedd676743aaa142fcd78650a998. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 21:08:19.545006 5380 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0d9ffedd676743aaa142fcd78650a998. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 21:08:19.546355 5118 tablet_service.cc:1467] Processing CreateTablet for tablet 0d9ffedd676743aaa142fcd78650a998 (DEFAULT_TABLE table=test-table [id=f440004f822d484894c3104918b34bdc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 21:08:19.546619 5118 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0d9ffedd676743aaa142fcd78650a998. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 21:08:19.547233 5460 tablet_bootstrap.cc:492] T 0d9ffedd676743aaa142fcd78650a998 P 8952ed545f774f86b20c86e7f5d26218: Bootstrap starting.
I20260924 21:08:19.547454 5461 tablet_bootstrap.cc:492] T 0d9ffedd676743aaa142fcd78650a998 P 6720c15219e04124b3ebc09c050bfa24: Bootstrap starting.
I20260924 21:08:19.547806 5460 tablet_bootstrap.cc:654] T 0d9ffedd676743aaa142fcd78650a998 P 8952ed545f774f86b20c86e7f5d26218: Neither blocks nor log segments found. Creating new log.
I20260924 21:08:19.548034 5461 tablet_bootstrap.cc:654] T 0d9ffedd676743aaa142fcd78650a998 P 6720c15219e04124b3ebc09c050bfa24: Neither blocks nor log segments found. Creating new log.
I20260924 21:08:19.548269 5460 log.cc:824] T 0d9ffedd676743aaa142fcd78650a998 P 8952ed545f774f86b20c86e7f5d26218: Log is configured to *not* fsync() on all Append() calls
I20260924 21:08:19.548355 5461 log.cc:824] T 0d9ffedd676743aaa142fcd78650a998 P 6720c15219e04124b3ebc09c050bfa24: Log is configured to *not* fsync() on all Append() calls
I20260924 21:08:19.548960 5460 tablet_bootstrap.cc:492] T 0d9ffedd676743aaa142fcd78650a998 P 8952ed545f774f86b20c86e7f5d26218: No bootstrap required, opened a new log
I20260924 21:08:19.549005 5460 ts_tablet_manager.cc:1397] T 0d9ffedd676743aaa142fcd78650a998 P 8952ed545f774f86b20c86e7f5d26218: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 21:08:19.549054 5461 tablet_bootstrap.cc:492] T 0d9ffedd676743aaa142fcd78650a998 P 6720c15219e04124b3ebc09c050bfa24: No bootstrap required, opened a new log
I20260924 21:08:19.549109 5461 ts_tablet_manager.cc:1397] T 0d9ffedd676743aaa142fcd78650a998 P 6720c15219e04124b3ebc09c050bfa24: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 21:08:19.549398 5464 tablet_bootstrap.cc:492] T 0d9ffedd676743aaa142fcd78650a998 P 9ec5861005e44c0ab8fca59defefb89b: Bootstrap starting.
I20260924 21:08:19.549995 5464 tablet_bootstrap.cc:654] T 0d9ffedd676743aaa142fcd78650a998 P 9ec5861005e44c0ab8fca59defefb89b: Neither blocks nor log segments found. Creating new log.
I20260924 21:08:19.550293 5464 log.cc:824] T 0d9ffedd676743aaa142fcd78650a998 P 9ec5861005e44c0ab8fca59defefb89b: Log is configured to *not* fsync() on all Append() calls
I20260924 21:08:19.551052 5464 tablet_bootstrap.cc:492] T 0d9ffedd676743aaa142fcd78650a998 P 9ec5861005e44c0ab8fca59defefb89b: No bootstrap required, opened a new log
I20260924 21:08:19.551098 5464 ts_tablet_manager.cc:1397] T 0d9ffedd676743aaa142fcd78650a998 P 9ec5861005e44c0ab8fca59defefb89b: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 21:08:19.551129 5461 raft_consensus.cc:348] T 0d9ffedd676743aaa142fcd78650a998 P 6720c15219e04124b3ebc09c050bfa24 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ec5861005e44c0ab8fca59defefb89b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34621 } } peers { permanent_uuid: "8952ed545f774f86b20c86e7f5d26218" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36319 } } peers { permanent_uuid: "6720c15219e04124b3ebc09c050bfa24" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43977 } }
I20260924 21:08:19.551286 5461 raft_consensus.cc:374] T 0d9ffedd676743aaa142fcd78650a998 P 6720c15219e04124b3ebc09c050bfa24 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 21:08:19.551308 5461 raft_consensus.cc:729] T 0d9ffedd676743aaa142fcd78650a998 P 6720c15219e04124b3ebc09c050bfa24 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6720c15219e04124b3ebc09c050bfa24, State: Initialized, Role: FOLLOWER
I20260924 21:08:19.551417 5461 consensus_queue.cc:260] T 0d9ffedd676743aaa142fcd78650a998 P 6720c15219e04124b3ebc09c050bfa24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ec5861005e44c0ab8fca59defefb89b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34621 } } peers { permanent_uuid: "8952ed545f774f86b20c86e7f5d26218" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36319 } } peers { permanent_uuid: "6720c15219e04124b3ebc09c050bfa24" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43977 } }
I20260924 21:08:19.551405 5460 raft_consensus.cc:348] T 0d9ffedd676743aaa142fcd78650a998 P 8952ed545f774f86b20c86e7f5d26218 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ec5861005e44c0ab8fca59defefb89b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34621 } } peers { permanent_uuid: "8952ed545f774f86b20c86e7f5d26218" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36319 } } peers { permanent_uuid: "6720c15219e04124b3ebc09c050bfa24" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43977 } }
I20260924 21:08:19.551543 5460 raft_consensus.cc:374] T 0d9ffedd676743aaa142fcd78650a998 P 8952ed545f774f86b20c86e7f5d26218 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 21:08:19.551573 5460 raft_consensus.cc:729] T 0d9ffedd676743aaa142fcd78650a998 P 8952ed545f774f86b20c86e7f5d26218 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8952ed545f774f86b20c86e7f5d26218, State: Initialized, Role: FOLLOWER
I20260924 21:08:19.551640 5315 heartbeater.cc:503] Master 127.0.98.62:37751 was elected leader, sending a full tablet report...
I20260924 21:08:19.551693 5460 consensus_queue.cc:260] T 0d9ffedd676743aaa142fcd78650a998 P 8952ed545f774f86b20c86e7f5d26218 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ec5861005e44c0ab8fca59defefb89b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34621 } } peers { permanent_uuid: "8952ed545f774f86b20c86e7f5d26218" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36319 } } peers { permanent_uuid: "6720c15219e04124b3ebc09c050bfa24" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43977 } }
I20260924 21:08:19.551914 5460 ts_tablet_manager.cc:1428] T 0d9ffedd676743aaa142fcd78650a998 P 8952ed545f774f86b20c86e7f5d26218: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 21:08:19.552012 5446 heartbeater.cc:503] Master 127.0.98.62:37751 was elected leader, sending a full tablet report...
I20260924 21:08:19.551638 5461 ts_tablet_manager.cc:1428] T 0d9ffedd676743aaa142fcd78650a998 P 6720c15219e04124b3ebc09c050bfa24: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260924 21:08:19.553539 5464 raft_consensus.cc:348] T 0d9ffedd676743aaa142fcd78650a998 P 9ec5861005e44c0ab8fca59defefb89b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ec5861005e44c0ab8fca59defefb89b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34621 } } peers { permanent_uuid: "8952ed545f774f86b20c86e7f5d26218" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36319 } } peers { permanent_uuid: "6720c15219e04124b3ebc09c050bfa24" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43977 } }
I20260924 21:08:19.553658 5464 raft_consensus.cc:374] T 0d9ffedd676743aaa142fcd78650a998 P 9ec5861005e44c0ab8fca59defefb89b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 21:08:19.553676 5464 raft_consensus.cc:729] T 0d9ffedd676743aaa142fcd78650a998 P 9ec5861005e44c0ab8fca59defefb89b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ec5861005e44c0ab8fca59defefb89b, State: Initialized, Role: FOLLOWER
I20260924 21:08:19.553789 5464 consensus_queue.cc:260] T 0d9ffedd676743aaa142fcd78650a998 P 9ec5861005e44c0ab8fca59defefb89b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ec5861005e44c0ab8fca59defefb89b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34621 } } peers { permanent_uuid: "8952ed545f774f86b20c86e7f5d26218" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36319 } } peers { permanent_uuid: "6720c15219e04124b3ebc09c050bfa24" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43977 } }
I20260924 21:08:19.554068 5464 ts_tablet_manager.cc:1428] T 0d9ffedd676743aaa142fcd78650a998 P 9ec5861005e44c0ab8fca59defefb89b: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 21:08:19.554085 5184 heartbeater.cc:503] Master 127.0.98.62:37751 was elected leader, sending a full tablet report...
W20260924 21:08:19.576782 5185 tablet.cc:2369] T 0d9ffedd676743aaa142fcd78650a998 P 9ec5861005e44c0ab8fca59defefb89b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 21:08:19.681103 5316 tablet.cc:2369] T 0d9ffedd676743aaa142fcd78650a998 P 6720c15219e04124b3ebc09c050bfa24: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 21:08:19.693465 5468 raft_consensus.cc:482] T 0d9ffedd676743aaa142fcd78650a998 P 9ec5861005e44c0ab8fca59defefb89b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 21:08:19.693570 5468 raft_consensus.cc:504] T 0d9ffedd676743aaa142fcd78650a998 P 9ec5861005e44c0ab8fca59defefb89b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ec5861005e44c0ab8fca59defefb89b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34621 } } peers { permanent_uuid: "8952ed545f774f86b20c86e7f5d26218" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36319 } } peers { permanent_uuid: "6720c15219e04124b3ebc09c050bfa24" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43977 } }
I20260924 21:08:19.693924 5468 leader_election.cc:288] T 0d9ffedd676743aaa142fcd78650a998 P 9ec5861005e44c0ab8fca59defefb89b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8952ed545f774f86b20c86e7f5d26218 (127.0.98.3:36319), 6720c15219e04124b3ebc09c050bfa24 (127.0.98.2:43977)
I20260924 21:08:19.695039 5458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:19.694156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:40041 (local address 127.0.98.3:36319)
0924 21:08:19.694198 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0924 21:08:19.694200 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 21:08:19.694209 (+ 9us) server_negotiation.cc:407] Connection header received
0924 21:08:19.694231 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:08:19.694232 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:08:19.694236 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:08:19.694288 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:08:19.694785 (+ 497us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 21:08:19.694787 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 21:08:19.694787 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 21:08:19.694791 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 21:08:19.694819 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 21:08:19.694823 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 21:08:19.694889 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0924 21:08:19.694953 (+ 64us) server_negotiation.cc:299] Negotiation successful
0924 21:08:19.694966 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260924 21:08:19.695032 5186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:19.694067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36319 (local address 127.0.98.1:40041)
0924 21:08:19.694115 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0924 21:08:19.694126 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 21:08:19.694165 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 21:08:19.694307 (+ 142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 21:08:19.694307 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 21:08:19.694309 (+ 2us) client_negotiation.cc:189] Negotiated authn=SASL
0924 21:08:19.694676 (+ 367us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 21:08:19.694676 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0924 21:08:19.694683 (+ 7us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 21:08:19.694693 (+ 10us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 21:08:19.694897 (+ 204us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 21:08:19.694897 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 21:08:19.694898 (+ 1us) client_negotiation.cc:769] Sending connection context
0924 21:08:19.694941 (+ 43us) client_negotiation.cc:240] Negotiation successful
0924 21:08:19.694962 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29}
I20260924 21:08:19.695091 5459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:19.694140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:52049 (local address 127.0.98.2:43977)
0924 21:08:19.694178 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0924 21:08:19.694180 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 21:08:19.694340 (+ 160us) server_negotiation.cc:407] Connection header received
0924 21:08:19.694367 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:08:19.694368 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:08:19.694372 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:08:19.694410 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:08:19.694880 (+ 470us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 21:08:19.694882 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 21:08:19.694882 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 21:08:19.694885 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0924 21:08:19.694906 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 21:08:19.694910 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 21:08:19.694973 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0924 21:08:19.695021 (+ 48us) server_negotiation.cc:299] Negotiation successful
0924 21:08:19.695031 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 21:08:19.695199 5469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:19.694075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43977 (local address 127.0.98.1:52049)
0924 21:08:19.694241 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0924 21:08:19.694255 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 21:08:19.694310 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 21:08:19.694429 (+ 119us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 21:08:19.694430 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 21:08:19.694432 (+ 2us) client_negotiation.cc:189] Negotiated authn=SASL
0924 21:08:19.694807 (+ 375us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 21:08:19.694807 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0924 21:08:19.694811 (+ 4us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 21:08:19.694818 (+ 7us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 21:08:19.694975 (+ 157us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 21:08:19.694976 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 21:08:19.694976 (+ 0us) client_negotiation.cc:769] Sending connection context
0924 21:08:19.695012 (+ 36us) client_negotiation.cc:240] Negotiation successful
0924 21:08:19.695021 (+ 9us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":154,"mutex_wait_us":2,"thread_start_us":94,"threads_started":1}
I20260924 21:08:19.695483 5400 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d9ffedd676743aaa142fcd78650a998" candidate_uuid: "9ec5861005e44c0ab8fca59defefb89b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8952ed545f774f86b20c86e7f5d26218" is_pre_election: true
I20260924 21:08:19.695490 5269 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d9ffedd676743aaa142fcd78650a998" candidate_uuid: "9ec5861005e44c0ab8fca59defefb89b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6720c15219e04124b3ebc09c050bfa24" is_pre_election: true
I20260924 21:08:19.695602 5400 raft_consensus.cc:2454] T 0d9ffedd676743aaa142fcd78650a998 P 8952ed545f774f86b20c86e7f5d26218 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9ec5861005e44c0ab8fca59defefb89b in term 0.
I20260924 21:08:19.695642 5269 raft_consensus.cc:2454] T 0d9ffedd676743aaa142fcd78650a998 P 6720c15219e04124b3ebc09c050bfa24 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9ec5861005e44c0ab8fca59defefb89b in term 0.
I20260924 21:08:19.695815 5073 leader_election.cc:302] T 0d9ffedd676743aaa142fcd78650a998 P 9ec5861005e44c0ab8fca59defefb89b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8952ed545f774f86b20c86e7f5d26218, 9ec5861005e44c0ab8fca59defefb89b; no voters:
I20260924 21:08:19.695953 5468 raft_consensus.cc:2781] T 0d9ffedd676743aaa142fcd78650a998 P 9ec5861005e44c0ab8fca59defefb89b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 21:08:19.696003 5468 raft_consensus.cc:482] T 0d9ffedd676743aaa142fcd78650a998 P 9ec5861005e44c0ab8fca59defefb89b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 21:08:19.696022 5468 raft_consensus.cc:3037] T 0d9ffedd676743aaa142fcd78650a998 P 9ec5861005e44c0ab8fca59defefb89b [term 0 FOLLOWER]: Advancing to term 1
I20260924 21:08:19.696627 5468 raft_consensus.cc:504] T 0d9ffedd676743aaa142fcd78650a998 P 9ec5861005e44c0ab8fca59defefb89b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ec5861005e44c0ab8fca59defefb89b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34621 } } peers { permanent_uuid: "8952ed545f774f86b20c86e7f5d26218" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36319 } } peers { permanent_uuid: "6720c15219e04124b3ebc09c050bfa24" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43977 } }
I20260924 21:08:19.696741 5468 leader_election.cc:288] T 0d9ffedd676743aaa142fcd78650a998 P 9ec5861005e44c0ab8fca59defefb89b [CANDIDATE]: Term 1 election: Requested vote from peers 8952ed545f774f86b20c86e7f5d26218 (127.0.98.3:36319), 6720c15219e04124b3ebc09c050bfa24 (127.0.98.2:43977)
I20260924 21:08:19.696880 5400 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d9ffedd676743aaa142fcd78650a998" candidate_uuid: "9ec5861005e44c0ab8fca59defefb89b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8952ed545f774f86b20c86e7f5d26218"
I20260924 21:08:19.696935 5400 raft_consensus.cc:3037] T 0d9ffedd676743aaa142fcd78650a998 P 8952ed545f774f86b20c86e7f5d26218 [term 0 FOLLOWER]: Advancing to term 1
I20260924 21:08:19.697556 5400 raft_consensus.cc:2454] T 0d9ffedd676743aaa142fcd78650a998 P 8952ed545f774f86b20c86e7f5d26218 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9ec5861005e44c0ab8fca59defefb89b in term 1.
I20260924 21:08:19.697656 5269 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d9ffedd676743aaa142fcd78650a998" candidate_uuid: "9ec5861005e44c0ab8fca59defefb89b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6720c15219e04124b3ebc09c050bfa24"
I20260924 21:08:19.697695 5073 leader_election.cc:302] T 0d9ffedd676743aaa142fcd78650a998 P 9ec5861005e44c0ab8fca59defefb89b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8952ed545f774f86b20c86e7f5d26218, 9ec5861005e44c0ab8fca59defefb89b; no voters:
I20260924 21:08:19.697726 5269 raft_consensus.cc:3037] T 0d9ffedd676743aaa142fcd78650a998 P 6720c15219e04124b3ebc09c050bfa24 [term 0 FOLLOWER]: Advancing to term 1
I20260924 21:08:19.697827 5468 raft_consensus.cc:2781] T 0d9ffedd676743aaa142fcd78650a998 P 9ec5861005e44c0ab8fca59defefb89b [term 1 FOLLOWER]: Leader election won for term 1
I20260924 21:08:19.697973 5468 raft_consensus.cc:686] T 0d9ffedd676743aaa142fcd78650a998 P 9ec5861005e44c0ab8fca59defefb89b [term 1 LEADER]: Becoming Leader. State: Replica: 9ec5861005e44c0ab8fca59defefb89b, State: Running, Role: LEADER
I20260924 21:08:19.698117 5468 consensus_queue.cc:237] T 0d9ffedd676743aaa142fcd78650a998 P 9ec5861005e44c0ab8fca59defefb89b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ec5861005e44c0ab8fca59defefb89b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34621 } } peers { permanent_uuid: "8952ed545f774f86b20c86e7f5d26218" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36319 } } peers { permanent_uuid: "6720c15219e04124b3ebc09c050bfa24" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43977 } }
I20260924 21:08:19.698611 5269 raft_consensus.cc:2454] T 0d9ffedd676743aaa142fcd78650a998 P 6720c15219e04124b3ebc09c050bfa24 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9ec5861005e44c0ab8fca59defefb89b in term 1.
I20260924 21:08:19.698866 4996 catalog_manager.cc:5495] T 0d9ffedd676743aaa142fcd78650a998 P 9ec5861005e44c0ab8fca59defefb89b reported cstate change: term changed from 0 to 1, leader changed from <none> to 9ec5861005e44c0ab8fca59defefb89b (127.0.98.1). New cstate: current_term: 1 leader_uuid: "9ec5861005e44c0ab8fca59defefb89b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ec5861005e44c0ab8fca59defefb89b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34621 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8952ed545f774f86b20c86e7f5d26218" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36319 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6720c15219e04124b3ebc09c050bfa24" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43977 } health_report { overall_health: UNKNOWN } } }
I20260924 21:08:19.724332 5456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:19.723423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48286 (local address 127.0.98.1:34621)
0924 21:08:19.723472 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0924 21:08:19.723473 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 21:08:19.723482 (+ 9us) server_negotiation.cc:407] Connection header received
0924 21:08:19.723546 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:08:19.723547 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:08:19.723561 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:08:19.723619 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:08:19.724113 (+ 494us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 21:08:19.724114 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 21:08:19.724115 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 21:08:19.724119 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 21:08:19.724131 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 21:08:19.724139 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 21:08:19.724192 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0924 21:08:19.724258 (+ 66us) server_negotiation.cc:299] Negotiation successful
0924 21:08:19.724271 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260924 21:08:19.725886 5269 raft_consensus.cc:1261] T 0d9ffedd676743aaa142fcd78650a998 P 6720c15219e04124b3ebc09c050bfa24 [term 1 FOLLOWER]: Refusing update from remote peer 9ec5861005e44c0ab8fca59defefb89b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 21:08:19.725926 5400 raft_consensus.cc:1261] T 0d9ffedd676743aaa142fcd78650a998 P 8952ed545f774f86b20c86e7f5d26218 [term 1 FOLLOWER]: Refusing update from remote peer 9ec5861005e44c0ab8fca59defefb89b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 21:08:19.726157 5470 consensus_queue.cc:1035] T 0d9ffedd676743aaa142fcd78650a998 P 9ec5861005e44c0ab8fca59defefb89b [LEADER]: Connected to new peer: Peer: permanent_uuid: "6720c15219e04124b3ebc09c050bfa24" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43977 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 21:08:19.726255 5468 consensus_queue.cc:1035] T 0d9ffedd676743aaa142fcd78650a998 P 9ec5861005e44c0ab8fca59defefb89b [LEADER]: Connected to new peer: Peer: permanent_uuid: "8952ed545f774f86b20c86e7f5d26218" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36319 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 21:08:19.728662 5475 mvcc.cc:205] Tried to move back new op lower bound from 7333003672474906624 to 7333003672364032000. Current Snapshot: MvccSnapshot[applied={T|T < 7333003672474906624}]
I20260924 21:08:19.730098 4996 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49908:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 21:08:19.730178 4996 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49908:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 21:08:19.730731 4996 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 924f420cede648c8b0e3311467393b0c: Sending DeleteTablet for 3 replicas of tablet 0d9ffedd676743aaa142fcd78650a998
I20260924 21:08:19.730995 5118 tablet_service.cc:1514] Processing DeleteTablet for tablet 0d9ffedd676743aaa142fcd78650a998 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 21:08:19 UTC) from {username='slave'} at 127.0.0.1:48274
I20260924 21:08:19.731031 5380 tablet_service.cc:1514] Processing DeleteTablet for tablet 0d9ffedd676743aaa142fcd78650a998 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 21:08:19 UTC) from {username='slave'} at 127.0.0.1:34020
I20260924 21:08:19.731175 5249 tablet_service.cc:1514] Processing DeleteTablet for tablet 0d9ffedd676743aaa142fcd78650a998 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 21:08:19 UTC) from {username='slave'} at 127.0.0.1:37124
I20260924 21:08:19.731220 5484 tablet_replica.cc:330] T 0d9ffedd676743aaa142fcd78650a998 P 8952ed545f774f86b20c86e7f5d26218: stopping tablet replica
I20260924 21:08:19.731297 5484 raft_consensus.cc:2229] T 0d9ffedd676743aaa142fcd78650a998 P 8952ed545f774f86b20c86e7f5d26218 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 21:08:19.731395 5484 raft_consensus.cc:2258] T 0d9ffedd676743aaa142fcd78650a998 P 8952ed545f774f86b20c86e7f5d26218 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 21:08:19.731544 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAzNNjS/build/release/bin/kudu with pid 5056
I20260924 21:08:19.731755 5484 ts_tablet_manager.cc:1905] T 0d9ffedd676743aaa142fcd78650a998 P 8952ed545f774f86b20c86e7f5d26218: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 21:08:19.733286 5484 ts_tablet_manager.cc:1918] T 0d9ffedd676743aaa142fcd78650a998 P 8952ed545f774f86b20c86e7f5d26218: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 21:08:19.733350 5484 log.cc:1196] T 0d9ffedd676743aaa142fcd78650a998 P 8952ed545f774f86b20c86e7f5d26218: Deleting WAL directory at /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790284076800039-392-0/minicluster-data/ts-2/wal/wals/0d9ffedd676743aaa142fcd78650a998
I20260924 21:08:19.733600 5484 ts_tablet_manager.cc:1939] T 0d9ffedd676743aaa142fcd78650a998 P 8952ed545f774f86b20c86e7f5d26218: Deleting consensus metadata
I20260924 21:08:19.733740 5485 tablet_replica.cc:330] T 0d9ffedd676743aaa142fcd78650a998 P 6720c15219e04124b3ebc09c050bfa24: stopping tablet replica
I20260924 21:08:19.733850 5485 raft_consensus.cc:2229] T 0d9ffedd676743aaa142fcd78650a998 P 6720c15219e04124b3ebc09c050bfa24 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 21:08:19.733956 5485 raft_consensus.cc:2258] T 0d9ffedd676743aaa142fcd78650a998 P 6720c15219e04124b3ebc09c050bfa24 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 21:08:19.734047 4982 catalog_manager.cc:4841] TS 8952ed545f774f86b20c86e7f5d26218 (127.0.98.3:36319): tablet 0d9ffedd676743aaa142fcd78650a998 (table test-table [id=f440004f822d484894c3104918b34bdc]) successfully deleted
I20260924 21:08:19.734295 5485 ts_tablet_manager.cc:1905] T 0d9ffedd676743aaa142fcd78650a998 P 6720c15219e04124b3ebc09c050bfa24: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 21:08:19.735882 5485 ts_tablet_manager.cc:1918] T 0d9ffedd676743aaa142fcd78650a998 P 6720c15219e04124b3ebc09c050bfa24: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 21:08:19.735946 5485 log.cc:1196] T 0d9ffedd676743aaa142fcd78650a998 P 6720c15219e04124b3ebc09c050bfa24: Deleting WAL directory at /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790284076800039-392-0/minicluster-data/ts-1/wal/wals/0d9ffedd676743aaa142fcd78650a998
I20260924 21:08:19.736150 5485 ts_tablet_manager.cc:1939] T 0d9ffedd676743aaa142fcd78650a998 P 6720c15219e04124b3ebc09c050bfa24: Deleting consensus metadata
I20260924 21:08:19.736466 4980 catalog_manager.cc:4841] TS 6720c15219e04124b3ebc09c050bfa24 (127.0.98.2:43977): tablet 0d9ffedd676743aaa142fcd78650a998 (table test-table [id=f440004f822d484894c3104918b34bdc]) successfully deleted
W20260924 21:08:19.738138 4982 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:34621 (error 108)
I20260924 21:08:19.738291 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAzNNjS/build/release/bin/kudu with pid 5187
I20260924 21:08:19.738521 5455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:19.738421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34621 (local address 127.0.0.1:48298)
0924 21:08:19.738461 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0924 21:08:19.738490 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:34621: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
W20260924 21:08:19.738608 4982 catalog_manager.cc:4584] TS 9ec5861005e44c0ab8fca59defefb89b (127.0.98.1:34621): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 0d9ffedd676743aaa142fcd78650a998: Network error: Client connection negotiation failed: client connection to 127.0.98.1:34621: connect: Connection refused (error 111)
I20260924 21:08:19.743000 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAzNNjS/build/release/bin/kudu with pid 5318
I20260924 21:08:19.746989 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAzNNjS/build/release/bin/kudu with pid 4965
2026-09-24T21:08:19Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (657 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260924 21:08:19.765164 392 mini_oidc.cc:152] Starting JWKS server
I20260924 21:08:19.766059 392 webserver.cc:460] Webserver started at https://127.0.0.1:45041/ using document root <none> and password file <none>
I20260924 21:08:19.766144 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260924 21:08:19.766299 392 webserver.cc:460] Webserver started at http://127.0.0.1:46717/ using document root <none> and password file <none>
2026-09-24T21:08:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T21:08:19Z Disabled control of system clock
I20260924 21:08:19.772997 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAzNNjS/build/release/bin/kudu
/tmp/dist-test-taskAzNNjS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790284076800039-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790284076800039-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790284076800039-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790284076800039-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42713
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:38287
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42713
--enable_jwt_token_auth=true
--jwks_url=https://localhost:45041/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790284076800039-392-0/testchainca.cert with env {}
W20260924 21:08:19.842386 5492 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 21:08:19.842547 5492 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 21:08:19.842566 5492 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 21:08:19.843487 5492 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 21:08:19.843514 5492 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 21:08:19.843525 5492 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 21:08:19.843535 5492 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 21:08:19.843546 5492 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260924 21:08:19.843556 5492 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260924 21:08:19.843573 5492 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:45041/jwks.json
I20260924 21:08:19.844797 5492 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38287
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790284076800039-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790284076800039-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42713
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42713
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:45041/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790284076800039-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790284076800039-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5492
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790284076800039-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 19:49:06 UTC on ccycloud.quasar-vhqjoc.root.comops.site
build id 4670864
I20260924 21:08:19.844946 5492 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 21:08:19.845072 5492 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 21:08:19.849063 5498 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 21:08:19.849092 5500 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 21:08:19.849092 5497 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 21:08:19.849237 5492 server_base.cc:1034] running on GCE node
I20260924 21:08:19.849481 5492 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 21:08:19.849709 5492 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 21:08:19.850849 5492 hybrid_clock.cc:648] HybridClock initialized: now 1790284099850834 us; error 27 us; skew 500 ppm
I20260924 21:08:19.851791 5492 webserver.cc:460] Webserver started at http://127.0.98.62:44743/ using document root <none> and password file <none>
I20260924 21:08:19.851991 5492 fs_manager.cc:359] Metadata directory not provided
I20260924 21:08:19.852046 5492 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 21:08:19.852147 5492 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 21:08:19.852943 5492 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790284076800039-392-0/minicluster-data/master-0/data/instance:
uuid: "b764166b34684b79baaa5f48c69dccdb"
format_stamp: "Formatted at 2026-09-24 21:08:19 on dist-test-slave-c7tk"
server_key: "af3f5a13c6ce62465637d76338bac518"
server_key_iv: "aa6d6f788671b01aa2ee0bbf63a9b172"
server_key_version: "encryptionkey@0"
I20260924 21:08:19.853253 5492 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790284076800039-392-0/minicluster-data/master-0/wal/instance:
uuid: "b764166b34684b79baaa5f48c69dccdb"
format_stamp: "Formatted at 2026-09-24 21:08:19 on dist-test-slave-c7tk"
server_key: "af3f5a13c6ce62465637d76338bac518"
server_key_iv: "aa6d6f788671b01aa2ee0bbf63a9b172"
server_key_version: "encryptionkey@0"
I20260924 21:08:19.854470 5492 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 21:08:19.855118 5506 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 21:08:19.855260 5492 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 21:08:19.855341 5492 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790284076800039-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790284076800039-392-0/minicluster-data/master-0/wal
uuid: "b764166b34684b79baaa5f48c69dccdb"
format_stamp: "Formatted at 2026-09-24 21:08:19 on dist-test-slave-c7tk"
server_key: "af3f5a13c6ce62465637d76338bac518"
server_key_iv: "aa6d6f788671b01aa2ee0bbf63a9b172"
server_key_version: "encryptionkey@0"
I20260924 21:08:19.855409 5492 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790284076800039-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790284076800039-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790284076800039-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 21:08:19.877121 5492 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 21:08:19.877329 5492 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 21:08:19.877445 5492 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 21:08:19.880625 5492 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42713
I20260924 21:08:19.880661 5558 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42713 every 8 connection(s)
I20260924 21:08:19.880913 5492 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790284076800039-392-0/minicluster-data/master-0/data/info.pb
I20260924 21:08:19.881660 5559 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 21:08:19.883466 5559 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b764166b34684b79baaa5f48c69dccdb: Bootstrap starting.
I20260924 21:08:19.883961 5559 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b764166b34684b79baaa5f48c69dccdb: Neither blocks nor log segments found. Creating new log.
I20260924 21:08:19.884204 5559 log.cc:824] T 00000000000000000000000000000000 P b764166b34684b79baaa5f48c69dccdb: Log is configured to *not* fsync() on all Append() calls
I20260924 21:08:19.884730 5559 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b764166b34684b79baaa5f48c69dccdb: No bootstrap required, opened a new log
I20260924 21:08:19.886005 5559 raft_consensus.cc:348] T 00000000000000000000000000000000 P b764166b34684b79baaa5f48c69dccdb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b764166b34684b79baaa5f48c69dccdb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42713 } }
I20260924 21:08:19.886132 5559 raft_consensus.cc:374] T 00000000000000000000000000000000 P b764166b34684b79baaa5f48c69dccdb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 21:08:19.886181 5559 raft_consensus.cc:729] T 00000000000000000000000000000000 P b764166b34684b79baaa5f48c69dccdb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b764166b34684b79baaa5f48c69dccdb, State: Initialized, Role: FOLLOWER
I20260924 21:08:19.886303 5559 consensus_queue.cc:260] T 00000000000000000000000000000000 P b764166b34684b79baaa5f48c69dccdb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b764166b34684b79baaa5f48c69dccdb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42713 } }
I20260924 21:08:19.886355 5559 raft_consensus.cc:388] T 00000000000000000000000000000000 P b764166b34684b79baaa5f48c69dccdb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 21:08:19.886387 5559 raft_consensus.cc:482] T 00000000000000000000000000000000 P b764166b34684b79baaa5f48c69dccdb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 21:08:19.886436 5559 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b764166b34684b79baaa5f48c69dccdb [term 0 FOLLOWER]: Advancing to term 1
I20260924 21:08:19.886793 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAzNNjS/build/release/bin/kudu as pid 5492
I20260924 21:08:19.886883 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790284076800039-392-0/minicluster-data/master-0/wal/instance
I20260924 21:08:19.886859 5559 raft_consensus.cc:504] T 00000000000000000000000000000000 P b764166b34684b79baaa5f48c69dccdb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b764166b34684b79baaa5f48c69dccdb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42713 } }
I20260924 21:08:19.886969 5559 leader_election.cc:302] T 00000000000000000000000000000000 P b764166b34684b79baaa5f48c69dccdb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b764166b34684b79baaa5f48c69dccdb; no voters:
I20260924 21:08:19.887113 392 external_mini_cluster.cc:1428] Setting key 85157039ece4486c7c1dfd491290ef32
I20260924 21:08:19.887217 5559 leader_election.cc:288] T 00000000000000000000000000000000 P b764166b34684b79baaa5f48c69dccdb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 21:08:19.887240 5562 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b764166b34684b79baaa5f48c69dccdb [term 1 FOLLOWER]: Leader election won for term 1
I20260924 21:08:19.887346 5562 raft_consensus.cc:686] T 00000000000000000000000000000000 P b764166b34684b79baaa5f48c69dccdb [term 1 LEADER]: Becoming Leader. State: Replica: b764166b34684b79baaa5f48c69dccdb, State: Running, Role: LEADER
I20260924 21:08:19.887425 5562 consensus_queue.cc:237] T 00000000000000000000000000000000 P b764166b34684b79baaa5f48c69dccdb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b764166b34684b79baaa5f48c69dccdb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42713 } }
I20260924 21:08:19.887658 5559 sys_catalog.cc:564] T 00000000000000000000000000000000 P b764166b34684b79baaa5f48c69dccdb [sys.catalog]: configured and running, proceeding with master startup.
I20260924 21:08:19.887714 5562 sys_catalog.cc:455] T 00000000000000000000000000000000 P b764166b34684b79baaa5f48c69dccdb [sys.catalog]: SysCatalogTable state changed. Reason: New leader b764166b34684b79baaa5f48c69dccdb. Latest consensus state: current_term: 1 leader_uuid: "b764166b34684b79baaa5f48c69dccdb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b764166b34684b79baaa5f48c69dccdb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42713 } } }
I20260924 21:08:19.887771 5562 sys_catalog.cc:458] T 00000000000000000000000000000000 P b764166b34684b79baaa5f48c69dccdb [sys.catalog]: This master's current role is: LEADER
I20260924 21:08:19.887822 5563 sys_catalog.cc:455] T 00000000000000000000000000000000 P b764166b34684b79baaa5f48c69dccdb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b764166b34684b79baaa5f48c69dccdb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b764166b34684b79baaa5f48c69dccdb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42713 } } }
I20260924 21:08:19.887903 5563 sys_catalog.cc:458] T 00000000000000000000000000000000 P b764166b34684b79baaa5f48c69dccdb [sys.catalog]: This master's current role is: LEADER
I20260924 21:08:19.887940 5568 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 21:08:19.888446 5568 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 21:08:19.890177 5568 catalog_manager.cc:1347] Generated new cluster ID: 81b39e1db9134e86985330d90beed6a3
I20260924 21:08:19.890225 5568 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 21:08:19.893102 5566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:19.887490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56320 (local address 127.0.98.62:42713)
0924 21:08:19.889134 (+ 1644us) server_negotiation.cc:206] Beginning negotiation
0924 21:08:19.889141 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 21:08:19.889172 (+ 31us) server_negotiation.cc:407] Connection header received
0924 21:08:19.889261 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:08:19.889267 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:08:19.889311 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:08:19.889534 (+ 223us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:08:19.889710 (+ 176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:19.890696 (+ 986us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:19.891467 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:19.891847 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:19.891972 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:19.892533 (+ 561us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 21:08:19.892536 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 21:08:19.892536 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 21:08:19.892697 (+ 161us) server_negotiation.cc:922] calling sasl_server_start()
0924 21:08:19.892724 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 21:08:19.892733 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 21:08:19.892799 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0924 21:08:19.892863 (+ 64us) server_negotiation.cc:299] Negotiation successful
0924 21:08:19.892937 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1576,"thread_start_us":57,"threads_started":1}
I20260924 21:08:19.893790 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 21:08:19.900584 5568 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 21:08:19.901139 5568 catalog_manager.cc:1504] Loading token signing keys...
I20260924 21:08:19.911690 5568 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b764166b34684b79baaa5f48c69dccdb: Generated new TSK 0
I20260924 21:08:19.911842 5568 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260924 21:08:19.935004 5589 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260924 21:08:19.935076 5589 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260924 21:08:19.935297 5566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:19.921663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56324 (local address 127.0.98.62:42713)
0924 21:08:19.921703 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0924 21:08:19.921705 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 21:08:19.921854 (+ 149us) server_negotiation.cc:407] Connection header received
0924 21:08:19.921958 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:08:19.921958 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:08:19.921968 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:08:19.922033 (+ 65us) server_negotiation.cc:226] Negotiated authn=JWT
0924 21:08:19.922141 (+ 108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:19.922806 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:19.923428 (+ 622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:19.923704 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:19.923768 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:19.923910 (+ 142us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0924 21:08:19.935112 (+ 11202us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45041/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0924 21:08:19.935246 (+ 134us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:56324: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45041/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":22}
W20260924 21:08:19.935338 5566 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:56324: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45041/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260924 21:08:19.935542 5588 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 21:08:19.921575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42713 (local address 127.0.0.1:56324)
0924 21:08:19.921763 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0924 21:08:19.921771 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0924 21:08:19.921879 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 21:08:19.922050 (+ 171us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 21:08:19.922050 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 21:08:19.922052 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0924 21:08:19.922096 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:19.922097 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:19.922872 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:19.922873 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:19.923362 (+ 489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:19.923363 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:19.923766 (+ 403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:19.923768 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:19.923805 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:19.923819 (+ 14us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0924 21:08:19.935216 (+ 11397us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45041/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0924 21:08:19.935289 (+ 73us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:42713: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45041/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":88,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.0.98.62:42713: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45041/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260924 21:08:19.935935 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAzNNjS/build/release/bin/kudu with pid 5492
2026-09-24T21:08:19Z chronyd exiting
I20260924 21:08:19.971024 392 test_util.cc:175] -----------------------------------------------
I20260924 21:08:19.971101 392 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790284076800039-392-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (206 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260924 21:08:19.971756 392 mini_oidc.cc:152] Starting JWKS server
I20260924 21:08:19.972344 392 webserver.cc:460] Webserver started at https://127.0.0.1:45247/ using document root <none> and password file <none>
I20260924 21:08:19.972401 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260924 21:08:19.972558 392 webserver.cc:460] Webserver started at http://127.0.0.1:43495/ using document root <none> and password file <none>
2026-09-24T21:08:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T21:08:19Z Disabled control of system clock
I20260924 21:08:19.978838 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAzNNjS/build/release/bin/kudu
/tmp/dist-test-taskAzNNjS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790284076800039-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790284076800039-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790284076800039-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790284076800039-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39171
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:37743
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39171
--enable_jwt_token_auth=true
--jwks_url=https://localhost:45247/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790284076800039-392-0/testchainca.cert with env {}
W20260924 21:08:20.052820 5595 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 21:08:20.052983 5595 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 21:08:20.053004 5595 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 21:08:20.054025 5595 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 21:08:20.054064 5595 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 21:08:20.054076 5595 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 21:08:20.054116 5595 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 21:08:20.054137 5595 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260924 21:08:20.054173 5595 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260924 21:08:20.054211 5595 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:45247/jwks.json
I20260924 21:08:20.055562 5595 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37743
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790284076800039-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790284076800039-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39171
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39171
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:45247/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790284076800039-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790284076800039-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5595
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790284076800039-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 19:49:06 UTC on ccycloud.quasar-vhqjoc.root.comops.site
build id 4670864
I20260924 21:08:20.055780 5595 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 21:08:20.055980 5595 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 21:08:20.060216 5601 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 21:08:20.060258 5603 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 21:08:20.060273 5600 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 21:08:20.060362 5595 server_base.cc:1034] running on GCE node
I20260924 21:08:20.060601 5595 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 21:08:20.060838 5595 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 21:08:20.061980 5595 hybrid_clock.cc:648] HybridClock initialized: now 1790284100061973 us; error 37 us; skew 500 ppm
I20260924 21:08:20.062958 5595 webserver.cc:460] Webserver started at http://127.0.98.62:33657/ using document root <none> and password file <none>
I20260924 21:08:20.063159 5595 fs_manager.cc:359] Metadata directory not provided
I20260924 21:08:20.063213 5595 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 21:08:20.063314 5595 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 21:08:20.064128 5595 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790284076800039-392-0/minicluster-data/master-0/data/instance:
uuid: "e70907d7522947cd9c6ae89cbd063652"
format_stamp: "Formatted at 2026-09-24 21:08:20 on dist-test-slave-c7tk"
server_key: "6ab201be145cd5f3bd12e194ea0fde79"
server_key_iv: "09bbc592d1b13a3978f90510dcca3fbe"
server_key_version: "encryptionkey@0"
I20260924 21:08:20.064443 5595 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790284076800039-392-0/minicluster-data/master-0/wal/instance:
uuid: "e70907d7522947cd9c6ae89cbd063652"
format_stamp: "Formatted at 2026-09-24 21:08:20 on dist-test-slave-c7tk"
server_key: "6ab201be145cd5f3bd12e194ea0fde79"
server_key_iv: "09bbc592d1b13a3978f90510dcca3fbe"
server_key_version: "encryptionkey@0"
I20260924 21:08:20.065516 5595 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 21:08:20.066174 5609 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 21:08:20.066308 5595 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 21:08:20.066392 5595 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790284076800039-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790284076800039-392-0/minicluster-data/master-0/wal
uuid: "e70907d7522947cd9c6ae89cbd063652"
format_stamp: "Formatted at 2026-09-24 21:08:20 on dist-test-slave-c7tk"
server_key: "6ab201be145cd5f3bd12e194ea0fde79"
server_key_iv: "09bbc592d1b13a3978f90510dcca3fbe"
server_key_version: "encryptionkey@0"
I20260924 21:08:20.066460 5595 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790284076800039-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790284076800039-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790284076800039-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 21:08:20.076457 5595 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 21:08:20.076658 5595 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 21:08:20.076773 5595 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 21:08:20.080056 5595 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39171
I20260924 21:08:20.080081 5661 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39171 every 8 connection(s)
I20260924 21:08:20.080324 5595 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790284076800039-392-0/minicluster-data/master-0/data/info.pb
I20260924 21:08:20.081058 5662 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 21:08:20.082087 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAzNNjS/build/release/bin/kudu as pid 5595
I20260924 21:08:20.082185 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790284076800039-392-0/minicluster-data/master-0/wal/instance
I20260924 21:08:20.082355 392 external_mini_cluster.cc:1428] Setting key 40982b943e76ffd99738cbbec025f453
I20260924 21:08:20.083209 5662 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e70907d7522947cd9c6ae89cbd063652: Bootstrap starting.
I20260924 21:08:20.083745 5662 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e70907d7522947cd9c6ae89cbd063652: Neither blocks nor log segments found. Creating new log.
I20260924 21:08:20.083979 5662 log.cc:824] T 00000000000000000000000000000000 P e70907d7522947cd9c6ae89cbd063652: Log is configured to *not* fsync() on all Append() calls
I20260924 21:08:20.084570 5662 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e70907d7522947cd9c6ae89cbd063652: No bootstrap required, opened a new log
I20260924 21:08:20.086447 5662 raft_consensus.cc:348] T 00000000000000000000000000000000 P e70907d7522947cd9c6ae89cbd063652 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e70907d7522947cd9c6ae89cbd063652" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39171 } }
I20260924 21:08:20.086596 5662 raft_consensus.cc:374] T 00000000000000000000000000000000 P e70907d7522947cd9c6ae89cbd063652 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 21:08:20.086648 5662 raft_consensus.cc:729] T 00000000000000000000000000000000 P e70907d7522947cd9c6ae89cbd063652 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e70907d7522947cd9c6ae89cbd063652, State: Initialized, Role: FOLLOWER
I20260924 21:08:20.086732 5665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:20.082714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35294 (local address 127.0.98.62:39171)
0924 21:08:20.082860 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0924 21:08:20.082876 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0924 21:08:20.082895 (+ 19us) server_negotiation.cc:407] Connection header received
0924 21:08:20.083194 (+ 299us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:08:20.083198 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:08:20.083242 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:08:20.083366 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:08:20.083556 (+ 190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:20.084318 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:20.085207 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:20.085526 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:20.085599 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:20.086170 (+ 571us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 21:08:20.086173 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 21:08:20.086173 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 21:08:20.086297 (+ 124us) server_negotiation.cc:922] calling sasl_server_start()
0924 21:08:20.086322 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 21:08:20.086331 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 21:08:20.086374 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0924 21:08:20.086455 (+ 81us) server_negotiation.cc:299] Negotiation successful
0924 21:08:20.086501 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89,"thread_start_us":68,"threads_started":1}
I20260924 21:08:20.086752 5662 consensus_queue.cc:260] T 00000000000000000000000000000000 P e70907d7522947cd9c6ae89cbd063652 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e70907d7522947cd9c6ae89cbd063652" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39171 } }
I20260924 21:08:20.086827 5662 raft_consensus.cc:388] T 00000000000000000000000000000000 P e70907d7522947cd9c6ae89cbd063652 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 21:08:20.086861 5662 raft_consensus.cc:482] T 00000000000000000000000000000000 P e70907d7522947cd9c6ae89cbd063652 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 21:08:20.086900 5662 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e70907d7522947cd9c6ae89cbd063652 [term 0 FOLLOWER]: Advancing to term 1
I20260924 21:08:20.087486 5662 raft_consensus.cc:504] T 00000000000000000000000000000000 P e70907d7522947cd9c6ae89cbd063652 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e70907d7522947cd9c6ae89cbd063652" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39171 } }
I20260924 21:08:20.087572 5662 leader_election.cc:302] T 00000000000000000000000000000000 P e70907d7522947cd9c6ae89cbd063652 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e70907d7522947cd9c6ae89cbd063652; no voters:
I20260924 21:08:20.087656 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 21:08:20.087708 5662 leader_election.cc:288] T 00000000000000000000000000000000 P e70907d7522947cd9c6ae89cbd063652 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 21:08:20.087775 5667 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e70907d7522947cd9c6ae89cbd063652 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 21:08:20.087881 5662 sys_catalog.cc:564] T 00000000000000000000000000000000 P e70907d7522947cd9c6ae89cbd063652 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 21:08:20.087954 5667 raft_consensus.cc:686] T 00000000000000000000000000000000 P e70907d7522947cd9c6ae89cbd063652 [term 1 LEADER]: Becoming Leader. State: Replica: e70907d7522947cd9c6ae89cbd063652, State: Running, Role: LEADER
I20260924 21:08:20.088094 5667 consensus_queue.cc:237] T 00000000000000000000000000000000 P e70907d7522947cd9c6ae89cbd063652 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e70907d7522947cd9c6ae89cbd063652" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39171 } }
I20260924 21:08:20.088486 5668 sys_catalog.cc:455] T 00000000000000000000000000000000 P e70907d7522947cd9c6ae89cbd063652 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e70907d7522947cd9c6ae89cbd063652" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e70907d7522947cd9c6ae89cbd063652" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39171 } } }
I20260924 21:08:20.088552 5668 sys_catalog.cc:458] T 00000000000000000000000000000000 P e70907d7522947cd9c6ae89cbd063652 [sys.catalog]: This master's current role is: LEADER
I20260924 21:08:20.088465 5667 sys_catalog.cc:455] T 00000000000000000000000000000000 P e70907d7522947cd9c6ae89cbd063652 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e70907d7522947cd9c6ae89cbd063652. Latest consensus state: current_term: 1 leader_uuid: "e70907d7522947cd9c6ae89cbd063652" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e70907d7522947cd9c6ae89cbd063652" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39171 } } }
I20260924 21:08:20.088663 5667 sys_catalog.cc:458] T 00000000000000000000000000000000 P e70907d7522947cd9c6ae89cbd063652 [sys.catalog]: This master's current role is: LEADER
I20260924 21:08:20.088980 5676 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 21:08:20.089460 5676 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 21:08:20.090816 5676 catalog_manager.cc:1347] Generated new cluster ID: e8c1f4498f294c9da5ec0f8858783d1f
I20260924 21:08:20.090865 5676 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 21:08:20.098227 5676 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 21:08:20.098659 5676 catalog_manager.cc:1504] Loading token signing keys...
I20260924 21:08:20.101301 5676 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e70907d7522947cd9c6ae89cbd063652: Generated new TSK 0
I20260924 21:08:20.101428 5676 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260924 21:08:20.126044 5692 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260924 21:08:20.126097 5692 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260924 21:08:20.126286 5665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:20.106487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35304 (local address 127.0.98.62:39171)
0924 21:08:20.106523 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0924 21:08:20.106524 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 21:08:20.106615 (+ 91us) server_negotiation.cc:407] Connection header received
0924 21:08:20.106672 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:08:20.106673 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:08:20.106681 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:08:20.106729 (+ 48us) server_negotiation.cc:226] Negotiated authn=JWT
0924 21:08:20.106829 (+ 100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:20.107224 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:20.108255 (+ 1031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:20.108434 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:20.108481 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:20.108636 (+ 155us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0924 21:08:20.126137 (+ 17501us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45247/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0924 21:08:20.126250 (+ 113us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:35304: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45247/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":21}
W20260924 21:08:20.126338 5665 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:35304: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45247/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260924 21:08:20.126389 5691 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 21:08:20.106417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39171 (local address 127.0.0.1:35304)
0924 21:08:20.106552 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0924 21:08:20.106562 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 21:08:20.106617 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 21:08:20.106731 (+ 114us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 21:08:20.106731 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 21:08:20.106733 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0924 21:08:20.106780 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:20.106781 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:20.107314 (+ 533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:20.107315 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:20.108183 (+ 868us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:20.108185 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:20.108496 (+ 311us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:20.108499 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:20.108556 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:20.108561 (+ 5us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0924 21:08:20.126231 (+ 17670us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45247/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0924 21:08:20.126310 (+ 79us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:39171: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45247/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":77,"threads_started":1}
I20260924 21:08:20.126609 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAzNNjS/build/release/bin/kudu with pid 5595
2026-09-24T21:08:20Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (191 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260924 21:08:20.163359 392 mini_oidc.cc:152] Starting JWKS server
I20260924 21:08:20.163857 392 webserver.cc:460] Webserver started at https://127.0.0.1:45245/ using document root <none> and password file <none>
I20260924 21:08:20.163897 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260924 21:08:20.164004 392 webserver.cc:460] Webserver started at http://127.0.0.1:41003/ using document root <none> and password file <none>
2026-09-24T21:08:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T21:08:20Z Disabled control of system clock
I20260924 21:08:20.170291 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAzNNjS/build/release/bin/kudu
/tmp/dist-test-taskAzNNjS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790284076800039-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790284076800039-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790284076800039-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790284076800039-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38505
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:39499
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38505
--enable_jwt_token_auth=true
--jwks_url=https://localhost:45245/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260924 21:08:20.239207 5698 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 21:08:20.239419 5698 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 21:08:20.239452 5698 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 21:08:20.240475 5698 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 21:08:20.240528 5698 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 21:08:20.240547 5698 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 21:08:20.240566 5698 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 21:08:20.240590 5698 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260924 21:08:20.240612 5698 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260924 21:08:20.240638 5698 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:45245/jwks.json
I20260924 21:08:20.241940 5698 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39499
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790284076800039-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790284076800039-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38505
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38505
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:45245/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790284076800039-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5698
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790284076800039-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 19:49:06 UTC on ccycloud.quasar-vhqjoc.root.comops.site
build id 4670864
I20260924 21:08:20.242172 5698 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 21:08:20.242424 5698 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 21:08:20.246537 5706 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 21:08:20.246538 5703 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 21:08:20.246582 5698 server_base.cc:1034] running on GCE node
W20260924 21:08:20.246544 5704 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 21:08:20.246945 5698 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 21:08:20.247221 5698 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 21:08:20.248347 5698 hybrid_clock.cc:648] HybridClock initialized: now 1790284100248329 us; error 33 us; skew 500 ppm
I20260924 21:08:20.249445 5698 webserver.cc:460] Webserver started at http://127.0.98.62:34701/ using document root <none> and password file <none>
I20260924 21:08:20.249641 5698 fs_manager.cc:359] Metadata directory not provided
I20260924 21:08:20.249678 5698 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 21:08:20.249823 5698 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 21:08:20.250633 5698 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790284076800039-392-0/minicluster-data/master-0/data/instance:
uuid: "b2217f4569cc43c1a607a92bab01b52b"
format_stamp: "Formatted at 2026-09-24 21:08:20 on dist-test-slave-c7tk"
server_key: "083bec490a558a21e409674adb089ac2"
server_key_iv: "b39afe0cf60383a61388433b70b7f518"
server_key_version: "encryptionkey@0"
I20260924 21:08:20.250921 5698 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790284076800039-392-0/minicluster-data/master-0/wal/instance:
uuid: "b2217f4569cc43c1a607a92bab01b52b"
format_stamp: "Formatted at 2026-09-24 21:08:20 on dist-test-slave-c7tk"
server_key: "083bec490a558a21e409674adb089ac2"
server_key_iv: "b39afe0cf60383a61388433b70b7f518"
server_key_version: "encryptionkey@0"
I20260924 21:08:20.251942 5698 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 21:08:20.252516 5712 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 21:08:20.252651 5698 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 21:08:20.252733 5698 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790284076800039-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790284076800039-392-0/minicluster-data/master-0/wal
uuid: "b2217f4569cc43c1a607a92bab01b52b"
format_stamp: "Formatted at 2026-09-24 21:08:20 on dist-test-slave-c7tk"
server_key: "083bec490a558a21e409674adb089ac2"
server_key_iv: "b39afe0cf60383a61388433b70b7f518"
server_key_version: "encryptionkey@0"
I20260924 21:08:20.252799 5698 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790284076800039-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790284076800039-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790284076800039-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 21:08:20.274279 5698 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 21:08:20.274506 5698 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 21:08:20.274629 5698 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 21:08:20.277835 5698 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38505
I20260924 21:08:20.277875 5764 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38505 every 8 connection(s)
I20260924 21:08:20.278123 5698 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790284076800039-392-0/minicluster-data/master-0/data/info.pb
I20260924 21:08:20.278703 5765 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 21:08:20.280718 5765 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b2217f4569cc43c1a607a92bab01b52b: Bootstrap starting.
I20260924 21:08:20.281169 5765 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b2217f4569cc43c1a607a92bab01b52b: Neither blocks nor log segments found. Creating new log.
I20260924 21:08:20.281414 5765 log.cc:824] T 00000000000000000000000000000000 P b2217f4569cc43c1a607a92bab01b52b: Log is configured to *not* fsync() on all Append() calls
I20260924 21:08:20.281916 5765 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b2217f4569cc43c1a607a92bab01b52b: No bootstrap required, opened a new log
I20260924 21:08:20.283174 5765 raft_consensus.cc:348] T 00000000000000000000000000000000 P b2217f4569cc43c1a607a92bab01b52b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2217f4569cc43c1a607a92bab01b52b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38505 } }
I20260924 21:08:20.283278 5765 raft_consensus.cc:374] T 00000000000000000000000000000000 P b2217f4569cc43c1a607a92bab01b52b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 21:08:20.283325 5765 raft_consensus.cc:729] T 00000000000000000000000000000000 P b2217f4569cc43c1a607a92bab01b52b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b2217f4569cc43c1a607a92bab01b52b, State: Initialized, Role: FOLLOWER
I20260924 21:08:20.283458 5765 consensus_queue.cc:260] T 00000000000000000000000000000000 P b2217f4569cc43c1a607a92bab01b52b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2217f4569cc43c1a607a92bab01b52b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38505 } }
I20260924 21:08:20.283510 5765 raft_consensus.cc:388] T 00000000000000000000000000000000 P b2217f4569cc43c1a607a92bab01b52b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 21:08:20.283540 5765 raft_consensus.cc:482] T 00000000000000000000000000000000 P b2217f4569cc43c1a607a92bab01b52b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 21:08:20.283581 5765 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b2217f4569cc43c1a607a92bab01b52b [term 0 FOLLOWER]: Advancing to term 1
I20260924 21:08:20.283769 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAzNNjS/build/release/bin/kudu as pid 5698
I20260924 21:08:20.283859 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790284076800039-392-0/minicluster-data/master-0/wal/instance
I20260924 21:08:20.284003 392 external_mini_cluster.cc:1428] Setting key 2211c663207fa00bce234d60f122b0e8
I20260924 21:08:20.284088 5765 raft_consensus.cc:504] T 00000000000000000000000000000000 P b2217f4569cc43c1a607a92bab01b52b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2217f4569cc43c1a607a92bab01b52b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38505 } }
I20260924 21:08:20.284178 5765 leader_election.cc:302] T 00000000000000000000000000000000 P b2217f4569cc43c1a607a92bab01b52b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b2217f4569cc43c1a607a92bab01b52b; no voters:
I20260924 21:08:20.284310 5765 leader_election.cc:288] T 00000000000000000000000000000000 P b2217f4569cc43c1a607a92bab01b52b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 21:08:20.284399 5768 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b2217f4569cc43c1a607a92bab01b52b [term 1 FOLLOWER]: Leader election won for term 1
I20260924 21:08:20.284538 5765 sys_catalog.cc:564] T 00000000000000000000000000000000 P b2217f4569cc43c1a607a92bab01b52b [sys.catalog]: configured and running, proceeding with master startup.
I20260924 21:08:20.284574 5768 raft_consensus.cc:686] T 00000000000000000000000000000000 P b2217f4569cc43c1a607a92bab01b52b [term 1 LEADER]: Becoming Leader. State: Replica: b2217f4569cc43c1a607a92bab01b52b, State: Running, Role: LEADER
I20260924 21:08:20.284637 5768 consensus_queue.cc:237] T 00000000000000000000000000000000 P b2217f4569cc43c1a607a92bab01b52b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2217f4569cc43c1a607a92bab01b52b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38505 } }
I20260924 21:08:20.284971 5772 sys_catalog.cc:455] T 00000000000000000000000000000000 P b2217f4569cc43c1a607a92bab01b52b [sys.catalog]: SysCatalogTable state changed. Reason: New leader b2217f4569cc43c1a607a92bab01b52b. Latest consensus state: current_term: 1 leader_uuid: "b2217f4569cc43c1a607a92bab01b52b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2217f4569cc43c1a607a92bab01b52b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38505 } } }
I20260924 21:08:20.285054 5772 sys_catalog.cc:458] T 00000000000000000000000000000000 P b2217f4569cc43c1a607a92bab01b52b [sys.catalog]: This master's current role is: LEADER
I20260924 21:08:20.285147 5771 sys_catalog.cc:455] T 00000000000000000000000000000000 P b2217f4569cc43c1a607a92bab01b52b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b2217f4569cc43c1a607a92bab01b52b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2217f4569cc43c1a607a92bab01b52b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38505 } } }
I20260924 21:08:20.285204 5771 sys_catalog.cc:458] T 00000000000000000000000000000000 P b2217f4569cc43c1a607a92bab01b52b [sys.catalog]: This master's current role is: LEADER
I20260924 21:08:20.285499 5779 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 21:08:20.285929 5779 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 21:08:20.287233 5779 catalog_manager.cc:1347] Generated new cluster ID: ed107980f00e42d1b0434f00a6a1ac1c
I20260924 21:08:20.287281 5779 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 21:08:20.289655 5770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:20.284314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50568 (local address 127.0.98.62:38505)
0924 21:08:20.284489 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0924 21:08:20.284497 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 21:08:20.285783 (+ 1286us) server_negotiation.cc:407] Connection header received
0924 21:08:20.285917 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:08:20.285923 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:08:20.285980 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:08:20.286079 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:08:20.286333 (+ 254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:20.287125 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:20.287868 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:20.288330 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:20.288472 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:20.289054 (+ 582us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 21:08:20.289057 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 21:08:20.289058 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 21:08:20.289217 (+ 159us) server_negotiation.cc:922] calling sasl_server_start()
0924 21:08:20.289245 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 21:08:20.289254 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 21:08:20.289354 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0924 21:08:20.289422 (+ 68us) server_negotiation.cc:299] Negotiation successful
0924 21:08:20.289489 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92,"thread_start_us":70,"threads_started":1}
I20260924 21:08:20.290477 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 21:08:20.298312 5779 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 21:08:20.298770 5779 catalog_manager.cc:1504] Loading token signing keys...
I20260924 21:08:20.304297 5779 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b2217f4569cc43c1a607a92bab01b52b: Generated new TSK 0
I20260924 21:08:20.304426 5779 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 21:08:20.314076 5770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:20.311184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50572 (local address 127.0.98.62:38505)
0924 21:08:20.311216 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0924 21:08:20.311218 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 21:08:20.311271 (+ 53us) server_negotiation.cc:407] Connection header received
0924 21:08:20.311315 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:08:20.311315 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:08:20.311325 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:08:20.311392 (+ 67us) server_negotiation.cc:226] Negotiated authn=JWT
0924 21:08:20.311460 (+ 68us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:20.312100 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:20.312683 (+ 583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:20.312939 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:20.312997 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:20.313108 (+ 111us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0924 21:08:20.313913 (+ 805us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45245/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0924 21:08:20.314024 (+ 111us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:50572: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45245/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":19}
W20260924 21:08:20.314121 5770 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:50572: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45245/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260924 21:08:20.314169 5794 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 21:08:20.311120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38505 (local address 127.0.0.1:50572)
0924 21:08:20.311215 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0924 21:08:20.311223 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0924 21:08:20.311274 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 21:08:20.311390 (+ 116us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 21:08:20.311391 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 21:08:20.311392 (+ 1us) client_negotiation.cc:189] Negotiated authn=JWT
0924 21:08:20.311427 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:20.311428 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:20.312159 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:20.312159 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:20.312622 (+ 463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:20.312623 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:20.312999 (+ 376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:20.313001 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:20.313033 (+ 32us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:20.313036 (+ 3us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0924 21:08:20.313992 (+ 956us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45245/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0924 21:08:20.314074 (+ 82us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:38505: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45245/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":76,"thread_start_us":60,"threads_started":1}
I20260924 21:08:20.314343 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAzNNjS/build/release/bin/kudu with pid 5698
E20260924 21:08:20.328249 5795 webserver.cc:533] Webserver: sslize failed:
E20260924 21:08:20.328305 5795 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-09-24T21:08:20Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (188 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-24T21:08:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T21:08:20Z Disabled control of system clock
I20260924 21:08:20.358529 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAzNNjS/build/release/bin/kudu
/tmp/dist-test-taskAzNNjS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790284076800039-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790284076800039-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790284076800039-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790284076800039-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40361
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:40439
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40361
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790284076800039-392-0/testchainca.cert with env {}
W20260924 21:08:20.429672 5799 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 21:08:20.429921 5799 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 21:08:20.429965 5799 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 21:08:20.431111 5799 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 21:08:20.431173 5799 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 21:08:20.431195 5799 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 21:08:20.431227 5799 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 21:08:20.431253 5799 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260924 21:08:20.431281 5799 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260924 21:08:20.431303 5799 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260924 21:08:20.432639 5799 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40439
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790284076800039-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790284076800039-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40361
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40361
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790284076800039-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790284076800039-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5799
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790284076800039-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 19:49:06 UTC on ccycloud.quasar-vhqjoc.root.comops.site
build id 4670864
I20260924 21:08:20.432888 5799 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 21:08:20.433081 5799 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 21:08:20.437217 5805 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 21:08:20.437280 5804 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 21:08:20.437368 5799 server_base.cc:1034] running on GCE node
W20260924 21:08:20.437424 5807 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 21:08:20.437626 5799 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 21:08:20.437937 5799 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 21:08:20.439092 5799 hybrid_clock.cc:648] HybridClock initialized: now 1790284100439081 us; error 24 us; skew 500 ppm
I20260924 21:08:20.443321 5799 webserver.cc:460] Webserver started at http://127.0.98.62:44985/ using document root <none> and password file <none>
I20260924 21:08:20.443522 5799 fs_manager.cc:359] Metadata directory not provided
I20260924 21:08:20.443562 5799 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 21:08:20.443681 5799 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 21:08:20.444514 5799 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790284076800039-392-0/minicluster-data/master-0/data/instance:
uuid: "7081c9e203d1422798bc1791b83f28fe"
format_stamp: "Formatted at 2026-09-24 21:08:20 on dist-test-slave-c7tk"
server_key: "e1b042a1c8d60149e86698c5e3562f8b"
server_key_iv: "bab28fb84817a68115721c14c73bc26f"
server_key_version: "encryptionkey@0"
I20260924 21:08:20.444790 5799 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790284076800039-392-0/minicluster-data/master-0/wal/instance:
uuid: "7081c9e203d1422798bc1791b83f28fe"
format_stamp: "Formatted at 2026-09-24 21:08:20 on dist-test-slave-c7tk"
server_key: "e1b042a1c8d60149e86698c5e3562f8b"
server_key_iv: "bab28fb84817a68115721c14c73bc26f"
server_key_version: "encryptionkey@0"
I20260924 21:08:20.445864 5799 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260924 21:08:20.446501 5813 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 21:08:20.446651 5799 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 21:08:20.446728 5799 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790284076800039-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790284076800039-392-0/minicluster-data/master-0/wal
uuid: "7081c9e203d1422798bc1791b83f28fe"
format_stamp: "Formatted at 2026-09-24 21:08:20 on dist-test-slave-c7tk"
server_key: "e1b042a1c8d60149e86698c5e3562f8b"
server_key_iv: "bab28fb84817a68115721c14c73bc26f"
server_key_version: "encryptionkey@0"
I20260924 21:08:20.446791 5799 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790284076800039-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790284076800039-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790284076800039-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 21:08:20.464457 5799 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 21:08:20.464681 5799 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 21:08:20.464807 5799 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 21:08:20.468360 5799 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40361
I20260924 21:08:20.468386 5865 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40361 every 8 connection(s)
I20260924 21:08:20.468670 5799 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790284076800039-392-0/minicluster-data/master-0/data/info.pb
I20260924 21:08:20.469498 5866 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 21:08:20.471336 5866 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7081c9e203d1422798bc1791b83f28fe: Bootstrap starting.
I20260924 21:08:20.471741 5866 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7081c9e203d1422798bc1791b83f28fe: Neither blocks nor log segments found. Creating new log.
I20260924 21:08:20.471938 5866 log.cc:824] T 00000000000000000000000000000000 P 7081c9e203d1422798bc1791b83f28fe: Log is configured to *not* fsync() on all Append() calls
I20260924 21:08:20.472447 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAzNNjS/build/release/bin/kudu as pid 5799
I20260924 21:08:20.472458 5866 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7081c9e203d1422798bc1791b83f28fe: No bootstrap required, opened a new log
I20260924 21:08:20.472548 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790284076800039-392-0/minicluster-data/master-0/wal/instance
I20260924 21:08:20.472709 392 external_mini_cluster.cc:1428] Setting key cb9a688be2fc2b63c24cb2efc97c05a1
I20260924 21:08:20.474207 5866 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7081c9e203d1422798bc1791b83f28fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7081c9e203d1422798bc1791b83f28fe" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40361 } }
I20260924 21:08:20.474356 5866 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7081c9e203d1422798bc1791b83f28fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 21:08:20.474398 5866 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7081c9e203d1422798bc1791b83f28fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7081c9e203d1422798bc1791b83f28fe, State: Initialized, Role: FOLLOWER
I20260924 21:08:20.474514 5866 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7081c9e203d1422798bc1791b83f28fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7081c9e203d1422798bc1791b83f28fe" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40361 } }
I20260924 21:08:20.474594 5866 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7081c9e203d1422798bc1791b83f28fe [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 21:08:20.474624 5866 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7081c9e203d1422798bc1791b83f28fe [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 21:08:20.474665 5866 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7081c9e203d1422798bc1791b83f28fe [term 0 FOLLOWER]: Advancing to term 1
I20260924 21:08:20.475096 5866 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7081c9e203d1422798bc1791b83f28fe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7081c9e203d1422798bc1791b83f28fe" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40361 } }
I20260924 21:08:20.475167 5866 leader_election.cc:302] T 00000000000000000000000000000000 P 7081c9e203d1422798bc1791b83f28fe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7081c9e203d1422798bc1791b83f28fe; no voters:
I20260924 21:08:20.475307 5866 leader_election.cc:288] T 00000000000000000000000000000000 P 7081c9e203d1422798bc1791b83f28fe [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 21:08:20.475330 5871 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7081c9e203d1422798bc1791b83f28fe [term 1 FOLLOWER]: Leader election won for term 1
I20260924 21:08:20.475419 5871 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7081c9e203d1422798bc1791b83f28fe [term 1 LEADER]: Becoming Leader. State: Replica: 7081c9e203d1422798bc1791b83f28fe, State: Running, Role: LEADER
I20260924 21:08:20.475485 5871 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7081c9e203d1422798bc1791b83f28fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7081c9e203d1422798bc1791b83f28fe" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40361 } }
I20260924 21:08:20.475514 5866 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7081c9e203d1422798bc1791b83f28fe [sys.catalog]: configured and running, proceeding with master startup.
I20260924 21:08:20.475765 5871 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7081c9e203d1422798bc1791b83f28fe [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7081c9e203d1422798bc1791b83f28fe. Latest consensus state: current_term: 1 leader_uuid: "7081c9e203d1422798bc1791b83f28fe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7081c9e203d1422798bc1791b83f28fe" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40361 } } }
I20260924 21:08:20.475862 5871 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7081c9e203d1422798bc1791b83f28fe [sys.catalog]: This master's current role is: LEADER
I20260924 21:08:20.475878 5872 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7081c9e203d1422798bc1791b83f28fe [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7081c9e203d1422798bc1791b83f28fe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7081c9e203d1422798bc1791b83f28fe" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40361 } } }
I20260924 21:08:20.475951 5872 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7081c9e203d1422798bc1791b83f28fe [sys.catalog]: This master's current role is: LEADER
I20260924 21:08:20.476639 5876 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 21:08:20.477136 5876 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 21:08:20.478519 5876 catalog_manager.cc:1347] Generated new cluster ID: 29d4a3065348437fa1ddd1d85cb9fd1b
I20260924 21:08:20.478570 5876 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 21:08:20.478979 5870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:20.473101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54170 (local address 127.0.98.62:40361)
0924 21:08:20.473310 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0924 21:08:20.473316 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 21:08:20.473336 (+ 20us) server_negotiation.cc:407] Connection header received
0924 21:08:20.474196 (+ 860us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:08:20.474201 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:08:20.474251 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:08:20.474402 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:08:20.474607 (+ 205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:20.475936 (+ 1329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:20.476955 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:20.477517 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:20.477635 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:20.478235 (+ 600us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 21:08:20.478238 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 21:08:20.478238 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 21:08:20.478388 (+ 150us) server_negotiation.cc:922] calling sasl_server_start()
0924 21:08:20.478443 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 21:08:20.478458 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 21:08:20.478550 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0924 21:08:20.478719 (+ 169us) server_negotiation.cc:299] Negotiation successful
0924 21:08:20.478789 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144,"thread_start_us":105,"threads_started":1}
I20260924 21:08:20.479722 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 21:08:20.511705 5876 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 21:08:20.512326 5876 catalog_manager.cc:1504] Loading token signing keys...
I20260924 21:08:20.516184 5876 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7081c9e203d1422798bc1791b83f28fe: Generated new TSK 0
I20260924 21:08:20.516337 5876 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 21:08:20.557446 5870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:20.523634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54172 (local address 127.0.98.62:40361)
0924 21:08:20.523690 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0924 21:08:20.523692 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 21:08:20.523731 (+ 39us) server_negotiation.cc:407] Connection header received
0924 21:08:20.523780 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:08:20.523781 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:08:20.523792 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:08:20.523859 (+ 67us) server_negotiation.cc:226] Negotiated authn=JWT
0924 21:08:20.523958 (+ 99us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:20.524585 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:20.525209 (+ 624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:20.525490 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:20.525556 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:20.525682 (+ 126us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0924 21:08:20.557199 (+ 31517us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0924 21:08:20.557370 (+ 171us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:54172: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":39}
W20260924 21:08:20.557515 5870 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:54172: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260924 21:08:20.557622 5895 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 21:08:20.523567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40361 (local address 127.0.0.1:54172)
0924 21:08:20.523686 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0924 21:08:20.523694 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0924 21:08:20.523740 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 21:08:20.523862 (+ 122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 21:08:20.523863 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 21:08:20.523864 (+ 1us) client_negotiation.cc:189] Negotiated authn=JWT
0924 21:08:20.523906 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:20.523908 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:20.524646 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:20.524646 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:20.525142 (+ 496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:20.525143 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:20.525564 (+ 421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:20.525567 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:20.525604 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:20.525608 (+ 4us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0924 21:08:20.557322 (+ 31714us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0924 21:08:20.557433 (+ 111us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:40361: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":70,"threads_started":1}
I20260924 21:08:20.557981 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAzNNjS/build/release/bin/kudu with pid 5799
2026-09-24T21:08:20Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (223 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (78 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (77 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790284076800039-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 21:08:20 dist-test-slave-c7tk krb5kdc[5907](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 21:08:20 dist-test-slave-c7tk krb5kdc[5907](info): set up 2 sockets
Sep 24 21:08:20 dist-test-slave-c7tk krb5kdc[5907](info): commencing operation
krb5kdc: starting...
W20260924 21:08:21.621670 392 mini_kdc.cc:121] Time spent starting KDC: real 0.891s user 0.000s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790284076800039-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790284076800039-392-0/krb5kdc/test-user.keytab.
Sep 24 21:08:21 dist-test-slave-c7tk krb5kdc[5907](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284101, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T21:08:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T21:08:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790284076800039-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790284076800039-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 21:08:21.721659 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAzNNjS/build/release/bin/kudu
/tmp/dist-test-taskAzNNjS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790284076800039-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790284076800039-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790284076800039-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790284076800039-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33183
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790284076800039-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34371
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33183
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790284076800039-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790284076800039-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790284076800039-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 21:08:21.793475 5920 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 21:08:21.793687 5920 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 21:08:21.793721 5920 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 21:08:21.794768 5920 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 21:08:21.794837 5920 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 21:08:21.794859 5920 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 21:08:21.794884 5920 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 21:08:21.794903 5920 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 21:08:21.796183 5920 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34371
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790284076800039-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790284076800039-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33183
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790284076800039-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33183
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790284076800039-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5920
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790284076800039-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 19:49:06 UTC on ccycloud.quasar-vhqjoc.root.comops.site
build id 4670864
I20260924 21:08:21.796402 5920 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 21:08:21.796586 5920 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 21:08:21.800635 5926 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 21:08:21.800647 5925 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 21:08:21.800639 5928 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 21:08:21.800827 5920 server_base.cc:1034] running on GCE node
I20260924 21:08:21.801020 5920 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 21:08:21.801318 5920 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 21:08:21.802452 5920 hybrid_clock.cc:648] HybridClock initialized: now 1790284101802440 us; error 22 us; skew 500 ppm
Sep 24 21:08:21 dist-test-slave-c7tk krb5kdc[5907](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284101, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 21:08:21.804529 5920 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 21:08:21.804908 5920 webserver.cc:460] Webserver started at http://127.0.98.62:37097/ using document root <none> and password file <none>
I20260924 21:08:21.805090 5920 fs_manager.cc:359] Metadata directory not provided
I20260924 21:08:21.805145 5920 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 21:08:21.805267 5920 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 21:08:21.806176 5920 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790284076800039-392-0/minicluster-data/master-0/data/instance:
uuid: "fc6611536e1e423aaad321bbbb3d5af9"
format_stamp: "Formatted at 2026-09-24 21:08:21 on dist-test-slave-c7tk"
server_key: "881e16f7f0443d575aebdcfada21d2e9"
server_key_iv: "07d6fe8046e7dcd1055afc0144718fed"
server_key_version: "encryptionkey@0"
I20260924 21:08:21.806474 5920 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790284076800039-392-0/minicluster-data/master-0/wal/instance:
uuid: "fc6611536e1e423aaad321bbbb3d5af9"
format_stamp: "Formatted at 2026-09-24 21:08:21 on dist-test-slave-c7tk"
server_key: "881e16f7f0443d575aebdcfada21d2e9"
server_key_iv: "07d6fe8046e7dcd1055afc0144718fed"
server_key_version: "encryptionkey@0"
I20260924 21:08:21.807513 5920 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 21:08:21.808074 5935 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 21:08:21.808207 5920 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 21:08:21.808303 5920 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790284076800039-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790284076800039-392-0/minicluster-data/master-0/wal
uuid: "fc6611536e1e423aaad321bbbb3d5af9"
format_stamp: "Formatted at 2026-09-24 21:08:21 on dist-test-slave-c7tk"
server_key: "881e16f7f0443d575aebdcfada21d2e9"
server_key_iv: "07d6fe8046e7dcd1055afc0144718fed"
server_key_version: "encryptionkey@0"
I20260924 21:08:21.808383 5920 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790284076800039-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790284076800039-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790284076800039-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 21:08:21.860661 5920 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 21:08:21.862174 5920 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 21:08:21.862309 5920 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 21:08:21.865768 5920 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33183
I20260924 21:08:21.865841 5987 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33183 every 8 connection(s)
I20260924 21:08:21.866247 5920 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790284076800039-392-0/minicluster-data/master-0/data/info.pb
I20260924 21:08:21.866778 5988 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 21:08:21.868666 5988 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fc6611536e1e423aaad321bbbb3d5af9: Bootstrap starting.
I20260924 21:08:21.869199 5988 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fc6611536e1e423aaad321bbbb3d5af9: Neither blocks nor log segments found. Creating new log.
I20260924 21:08:21.869467 5988 log.cc:824] T 00000000000000000000000000000000 P fc6611536e1e423aaad321bbbb3d5af9: Log is configured to *not* fsync() on all Append() calls
I20260924 21:08:21.870071 5988 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fc6611536e1e423aaad321bbbb3d5af9: No bootstrap required, opened a new log
I20260924 21:08:21.871444 5988 raft_consensus.cc:348] T 00000000000000000000000000000000 P fc6611536e1e423aaad321bbbb3d5af9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc6611536e1e423aaad321bbbb3d5af9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33183 } }
I20260924 21:08:21.871572 5988 raft_consensus.cc:374] T 00000000000000000000000000000000 P fc6611536e1e423aaad321bbbb3d5af9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 21:08:21.871622 5988 raft_consensus.cc:729] T 00000000000000000000000000000000 P fc6611536e1e423aaad321bbbb3d5af9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc6611536e1e423aaad321bbbb3d5af9, State: Initialized, Role: FOLLOWER
I20260924 21:08:21.871775 5988 consensus_queue.cc:260] T 00000000000000000000000000000000 P fc6611536e1e423aaad321bbbb3d5af9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc6611536e1e423aaad321bbbb3d5af9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33183 } }
I20260924 21:08:21.871858 5988 raft_consensus.cc:388] T 00000000000000000000000000000000 P fc6611536e1e423aaad321bbbb3d5af9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 21:08:21.871910 5988 raft_consensus.cc:482] T 00000000000000000000000000000000 P fc6611536e1e423aaad321bbbb3d5af9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 21:08:21.871956 5988 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fc6611536e1e423aaad321bbbb3d5af9 [term 0 FOLLOWER]: Advancing to term 1
I20260924 21:08:21.872704 5988 raft_consensus.cc:504] T 00000000000000000000000000000000 P fc6611536e1e423aaad321bbbb3d5af9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc6611536e1e423aaad321bbbb3d5af9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33183 } }
I20260924 21:08:21.872795 5988 leader_election.cc:302] T 00000000000000000000000000000000 P fc6611536e1e423aaad321bbbb3d5af9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fc6611536e1e423aaad321bbbb3d5af9; no voters:
I20260924 21:08:21.872957 5988 leader_election.cc:288] T 00000000000000000000000000000000 P fc6611536e1e423aaad321bbbb3d5af9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 21:08:21.873001 5991 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fc6611536e1e423aaad321bbbb3d5af9 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 21:08:21.873155 5988 sys_catalog.cc:564] T 00000000000000000000000000000000 P fc6611536e1e423aaad321bbbb3d5af9 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 21:08:21.873951 5991 raft_consensus.cc:686] T 00000000000000000000000000000000 P fc6611536e1e423aaad321bbbb3d5af9 [term 1 LEADER]: Becoming Leader. State: Replica: fc6611536e1e423aaad321bbbb3d5af9, State: Running, Role: LEADER
I20260924 21:08:21.874066 5991 consensus_queue.cc:237] T 00000000000000000000000000000000 P fc6611536e1e423aaad321bbbb3d5af9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc6611536e1e423aaad321bbbb3d5af9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33183 } }
I20260924 21:08:21.874542 5992 sys_catalog.cc:455] T 00000000000000000000000000000000 P fc6611536e1e423aaad321bbbb3d5af9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fc6611536e1e423aaad321bbbb3d5af9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc6611536e1e423aaad321bbbb3d5af9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33183 } } }
I20260924 21:08:21.874644 5992 sys_catalog.cc:458] T 00000000000000000000000000000000 P fc6611536e1e423aaad321bbbb3d5af9 [sys.catalog]: This master's current role is: LEADER
I20260924 21:08:21.874854 6005 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 21:08:21.875059 5994 sys_catalog.cc:455] T 00000000000000000000000000000000 P fc6611536e1e423aaad321bbbb3d5af9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fc6611536e1e423aaad321bbbb3d5af9. Latest consensus state: current_term: 1 leader_uuid: "fc6611536e1e423aaad321bbbb3d5af9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc6611536e1e423aaad321bbbb3d5af9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33183 } } }
I20260924 21:08:21.875130 5994 sys_catalog.cc:458] T 00000000000000000000000000000000 P fc6611536e1e423aaad321bbbb3d5af9 [sys.catalog]: This master's current role is: LEADER
I20260924 21:08:21.875355 6005 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 21:08:21.875903 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAzNNjS/build/release/bin/kudu as pid 5920
I20260924 21:08:21.875995 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790284076800039-392-0/minicluster-data/master-0/wal/instance
I20260924 21:08:21.876157 392 external_mini_cluster.cc:1428] Setting key a2343cddda6e177d70c1f6d0f00bf8c3
I20260924 21:08:21.876590 6005 catalog_manager.cc:1347] Generated new cluster ID: 9d5373387bab48f0bdfa2e0606644457
I20260924 21:08:21.876636 6005 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 24 21:08:21 dist-test-slave-c7tk krb5kdc[5907](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284101, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 21:08:21.883399 6005 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 21:08:21.883982 6005 catalog_manager.cc:1504] Loading token signing keys...
I20260924 21:08:21.885281 6011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:21.876508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33694 (local address 127.0.98.62:33183)
0924 21:08:21.876648 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0924 21:08:21.876655 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 21:08:21.876824 (+ 169us) server_negotiation.cc:407] Connection header received
0924 21:08:21.876944 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:08:21.876948 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:08:21.876982 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:08:21.877085 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:08:21.877606 (+ 521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:21.878507 (+ 901us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:21.879237 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:21.879660 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:21.879770 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:21.881735 (+ 1965us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 21:08:21.881738 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 21:08:21.881739 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 21:08:21.881749 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0924 21:08:21.883718 (+ 1969us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:08:21.884113 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:08:21.884116 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:08:21.884116 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:08:21.884193 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:08:21.884335 (+ 142us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:08:21.884335 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:08:21.884335 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:08:21.884523 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 21:08:21.884635 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0924 21:08:21.884975 (+ 340us) server_negotiation.cc:299] Negotiation successful
0924 21:08:21.885088 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109,"thread_start_us":55,"threads_started":1}
I20260924 21:08:21.895394 6005 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fc6611536e1e423aaad321bbbb3d5af9: Generated new TSK 0
I20260924 21:08:21.895527 6005 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790284076800039-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790284076800039-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260924 21:08:21.920323 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAzNNjS/build/release/bin/kudu
/tmp/dist-test-taskAzNNjS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790284076800039-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790284076800039-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790284076800039-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790284076800039-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33183
--keytab_file=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790284076800039-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34371
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790284076800039-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790284076800039-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790284076800039-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 21:08:21.996658 6014 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 21:08:21.996882 6014 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 21:08:21.996923 6014 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 21:08:21.998065 6014 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 21:08:21.998129 6014 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 21:08:21.998174 6014 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 21:08:21.999607 6014 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34371
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790284076800039-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790284076800039-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790284076800039-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790284076800039-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33183
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6014
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790284076800039-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 19:49:06 UTC on ccycloud.quasar-vhqjoc.root.comops.site
build id 4670864
I20260924 21:08:21.999866 6014 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 21:08:22.000077 6014 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 21:08:22.004696 6022 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 21:08:22.004729 6020 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 21:08:22.004870 6014 server_base.cc:1034] running on GCE node
W20260924 21:08:22.004740 6019 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 21:08:22.005132 6014 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 21:08:22.005342 6014 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 21:08:22.006469 6014 hybrid_clock.cc:648] HybridClock initialized: now 1790284102006456 us; error 26 us; skew 500 ppm
Sep 24 21:08:22 dist-test-slave-c7tk krb5kdc[5907](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284102, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 21:08:22.008859 6014 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 21:08:22.009291 6014 webserver.cc:460] Webserver started at http://127.0.98.1:38459/ using document root <none> and password file <none>
I20260924 21:08:22.009511 6014 fs_manager.cc:359] Metadata directory not provided
I20260924 21:08:22.009549 6014 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 21:08:22.009693 6014 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 21:08:22.010645 6014 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790284076800039-392-0/minicluster-data/ts-0/data/instance:
uuid: "372d6cf4552d4abcb5b392d5965f5f40"
format_stamp: "Formatted at 2026-09-24 21:08:22 on dist-test-slave-c7tk"
server_key: "aa00f5b0242722eb2c0a8637ac389641"
server_key_iv: "cc334d7a9e7e56aac5e4e67e71968813"
server_key_version: "encryptionkey@0"
I20260924 21:08:22.010977 6014 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790284076800039-392-0/minicluster-data/ts-0/wal/instance:
uuid: "372d6cf4552d4abcb5b392d5965f5f40"
format_stamp: "Formatted at 2026-09-24 21:08:22 on dist-test-slave-c7tk"
server_key: "aa00f5b0242722eb2c0a8637ac389641"
server_key_iv: "cc334d7a9e7e56aac5e4e67e71968813"
server_key_version: "encryptionkey@0"
I20260924 21:08:22.012162 6014 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 21:08:22.012778 6029 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 21:08:22.012924 6014 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 21:08:22.012993 6014 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790284076800039-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790284076800039-392-0/minicluster-data/ts-0/wal
uuid: "372d6cf4552d4abcb5b392d5965f5f40"
format_stamp: "Formatted at 2026-09-24 21:08:22 on dist-test-slave-c7tk"
server_key: "aa00f5b0242722eb2c0a8637ac389641"
server_key_iv: "cc334d7a9e7e56aac5e4e67e71968813"
server_key_version: "encryptionkey@0"
I20260924 21:08:22.013038 6014 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790284076800039-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790284076800039-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790284076800039-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 21:08:22.027357 6014 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 21:08:22.028901 6014 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 21:08:22.028996 6014 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 21:08:22.029137 6014 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 21:08:22.029418 6014 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 21:08:22.029450 6014 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 21:08:22.029470 6014 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 21:08:22.029528 6014 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 21:08:22.034869 6014 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:41109
I20260924 21:08:22.034936 6142 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:41109 every 8 connection(s)
I20260924 21:08:22.035195 6014 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790284076800039-392-0/minicluster-data/ts-0/data/info.pb
Sep 24 21:08:22 dist-test-slave-c7tk krb5kdc[5907](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284102, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 21:08:22.044319 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAzNNjS/build/release/bin/kudu as pid 6014
I20260924 21:08:22.044401 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790284076800039-392-0/minicluster-data/ts-0/wal/instance
I20260924 21:08:22.044592 392 external_mini_cluster.cc:1428] Setting key 802adf9a0e0d08c10620ac1d8612bc6b
I20260924 21:08:22.044898 6145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:22.035773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33183 (local address 127.0.98.1:49713)
0924 21:08:22.035926 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0924 21:08:22.035963 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0924 21:08:22.037080 (+ 1117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 21:08:22.037355 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 21:08:22.037359 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 21:08:22.037611 (+ 252us) client_negotiation.cc:189] Negotiated authn=SASL
0924 21:08:22.037863 (+ 252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:22.037868 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:22.038628 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:22.038629 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:22.039426 (+ 797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:22.039428 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:22.039906 (+ 478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:22.039907 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:22.040045 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:22.040486 (+ 441us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 21:08:22.040495 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 21:08:22.042029 (+ 1534us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 21:08:22.043529 (+ 1500us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 21:08:22.043534 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 21:08:22.043536 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 21:08:22.043736 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 21:08:22.043934 (+ 198us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 21:08:22.043935 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 21:08:22.043935 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 21:08:22.044039 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 21:08:22.044337 (+ 298us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 21:08:22.044345 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 21:08:22.044477 (+ 132us) client_negotiation.cc:769] Sending connection context
0924 21:08:22.044586 (+ 109us) client_negotiation.cc:240] Negotiation successful
0924 21:08:22.044735 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":73,"threads_started":1}
I20260924 21:08:22.045632 6011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:22.035816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:49713 (local address 127.0.98.62:33183)
0924 21:08:22.035907 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0924 21:08:22.035910 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 21:08:22.036039 (+ 129us) server_negotiation.cc:407] Connection header received
0924 21:08:22.037215 (+ 1176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:08:22.037217 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:08:22.037231 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:08:22.037302 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:08:22.037975 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:22.038510 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:22.039560 (+ 1050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:22.039758 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:22.039839 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:22.042121 (+ 2282us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 21:08:22.042123 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 21:08:22.042123 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 21:08:22.042131 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 21:08:22.043438 (+ 1307us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:08:22.043817 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:08:22.043818 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:08:22.043818 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:08:22.043875 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:08:22.044099 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:08:22.044100 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:08:22.044100 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:08:22.044252 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 21:08:22.044314 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0924 21:08:22.045423 (+ 1109us) server_negotiation.cc:299] Negotiation successful
0924 21:08:22.045511 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260924 21:08:22.045986 6143 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33183
I20260924 21:08:22.046078 6143 heartbeater.cc:464] Registering TS with master...
I20260924 21:08:22.046269 6143 heartbeater.cc:511] Master 127.0.98.62:33183 requested a full tablet report, sending...
I20260924 21:08:22.046804 5952 ts_manager.cc:194] Registered new tserver with Master: 372d6cf4552d4abcb5b392d5965f5f40 (127.0.98.1:41109)
I20260924 21:08:22.047588 5952 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:49713
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790284076800039-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790284076800039-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260924 21:08:22.069065 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAzNNjS/build/release/bin/kudu
/tmp/dist-test-taskAzNNjS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790284076800039-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790284076800039-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790284076800039-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790284076800039-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33183
--keytab_file=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790284076800039-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34371
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790284076800039-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790284076800039-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790284076800039-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 21:08:22.139593 6148 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 21:08:22.139776 6148 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 21:08:22.139807 6148 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 21:08:22.140779 6148 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 21:08:22.140834 6148 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 21:08:22.140882 6148 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 21:08:22.142225 6148 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34371
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790284076800039-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790284076800039-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790284076800039-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790284076800039-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33183
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6148
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790284076800039-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 19:49:06 UTC on ccycloud.quasar-vhqjoc.root.comops.site
build id 4670864
I20260924 21:08:22.142462 6148 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 21:08:22.142654 6148 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 21:08:22.146582 6153 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 21:08:22.146600 6156 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 21:08:22.146651 6154 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 21:08:22.146780 6148 server_base.cc:1034] running on GCE node
I20260924 21:08:22.146971 6148 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 21:08:22.147174 6148 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 21:08:22.148316 6148 hybrid_clock.cc:648] HybridClock initialized: now 1790284102148312 us; error 33 us; skew 500 ppm
Sep 24 21:08:22 dist-test-slave-c7tk krb5kdc[5907](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284102, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 21:08:22.150676 6148 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 21:08:22.151050 6148 webserver.cc:460] Webserver started at http://127.0.98.2:43895/ using document root <none> and password file <none>
I20260924 21:08:22.151242 6148 fs_manager.cc:359] Metadata directory not provided
I20260924 21:08:22.151301 6148 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 21:08:22.151415 6148 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 21:08:22.152343 6148 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790284076800039-392-0/minicluster-data/ts-1/data/instance:
uuid: "885b6061909c4ac785f2d028354ebd00"
format_stamp: "Formatted at 2026-09-24 21:08:22 on dist-test-slave-c7tk"
server_key: "754b6837fa621f118b5bac47b2dc5baf"
server_key_iv: "a382edaad1aa3646ad2e20d879e65993"
server_key_version: "encryptionkey@0"
I20260924 21:08:22.152725 6148 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790284076800039-392-0/minicluster-data/ts-1/wal/instance:
uuid: "885b6061909c4ac785f2d028354ebd00"
format_stamp: "Formatted at 2026-09-24 21:08:22 on dist-test-slave-c7tk"
server_key: "754b6837fa621f118b5bac47b2dc5baf"
server_key_iv: "a382edaad1aa3646ad2e20d879e65993"
server_key_version: "encryptionkey@0"
I20260924 21:08:22.153846 6148 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 21:08:22.154498 6163 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 21:08:22.154789 6148 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 21:08:22.154879 6148 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790284076800039-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790284076800039-392-0/minicluster-data/ts-1/wal
uuid: "885b6061909c4ac785f2d028354ebd00"
format_stamp: "Formatted at 2026-09-24 21:08:22 on dist-test-slave-c7tk"
server_key: "754b6837fa621f118b5bac47b2dc5baf"
server_key_iv: "a382edaad1aa3646ad2e20d879e65993"
server_key_version: "encryptionkey@0"
I20260924 21:08:22.154946 6148 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790284076800039-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790284076800039-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790284076800039-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 21:08:22.161074 6148 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 21:08:22.162627 6148 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 21:08:22.162765 6148 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 21:08:22.162952 6148 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 21:08:22.163240 6148 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 21:08:22.163269 6148 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 21:08:22.163286 6148 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 21:08:22.163328 6148 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 21:08:22.168217 6148 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:41305
I20260924 21:08:22.168284 6276 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:41305 every 8 connection(s)
I20260924 21:08:22.168594 6148 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790284076800039-392-0/minicluster-data/ts-1/data/info.pb
I20260924 21:08:22.173020 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAzNNjS/build/release/bin/kudu as pid 6148
I20260924 21:08:22.173125 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790284076800039-392-0/minicluster-data/ts-1/wal/instance
I20260924 21:08:22.173281 392 external_mini_cluster.cc:1428] Setting key 5f61421dd048353ba171866d98f67185
Sep 24 21:08:22 dist-test-slave-c7tk krb5kdc[5907](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284102, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 21:08:22.177377 6011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:22.169249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:58067 (local address 127.0.98.62:33183)
0924 21:08:22.169291 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0924 21:08:22.169293 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 21:08:22.169464 (+ 171us) server_negotiation.cc:407] Connection header received
0924 21:08:22.170358 (+ 894us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:08:22.170359 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:08:22.170375 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:08:22.170419 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:08:22.170946 (+ 527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:22.171419 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:22.172288 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:22.172480 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:22.172565 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:22.174850 (+ 2285us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 21:08:22.174851 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 21:08:22.174851 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 21:08:22.174866 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0924 21:08:22.176109 (+ 1243us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:08:22.176477 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:08:22.176478 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:08:22.176479 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:08:22.176530 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:08:22.176800 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:08:22.176801 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:08:22.176801 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:08:22.176908 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 21:08:22.176976 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0924 21:08:22.177210 (+ 234us) server_negotiation.cc:299] Negotiation successful
0924 21:08:22.177284 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 21:08:22.177536 6279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:22.169213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33183 (local address 127.0.98.2:58067)
0924 21:08:22.169343 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0924 21:08:22.169388 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0924 21:08:22.170261 (+ 873us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 21:08:22.170482 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 21:08:22.170486 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 21:08:22.170695 (+ 209us) client_negotiation.cc:189] Negotiated authn=SASL
0924 21:08:22.170840 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:22.170843 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:22.171500 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:22.171501 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:22.172192 (+ 691us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:22.172193 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:22.172573 (+ 380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:22.172574 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:22.172677 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:22.173170 (+ 493us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 21:08:22.173177 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0924 21:08:22.174764 (+ 1587us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 21:08:22.176202 (+ 1438us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 21:08:22.176207 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 21:08:22.176209 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 21:08:22.176397 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 21:08:22.176622 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 21:08:22.176623 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 21:08:22.176623 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 21:08:22.176729 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 21:08:22.176969 (+ 240us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 21:08:22.176974 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 21:08:22.177095 (+ 121us) client_negotiation.cc:769] Sending connection context
0924 21:08:22.177192 (+ 97us) client_negotiation.cc:240] Negotiation successful
0924 21:08:22.177323 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":64,"threads_started":1}
I20260924 21:08:22.177995 6277 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33183
I20260924 21:08:22.178090 6277 heartbeater.cc:464] Registering TS with master...
I20260924 21:08:22.178267 6277 heartbeater.cc:511] Master 127.0.98.62:33183 requested a full tablet report, sending...
I20260924 21:08:22.178643 5952 ts_manager.cc:194] Registered new tserver with Master: 885b6061909c4ac785f2d028354ebd00 (127.0.98.2:41305)
I20260924 21:08:22.179024 5952 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:58067
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790284076800039-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790284076800039-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260924 21:08:22.194710 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAzNNjS/build/release/bin/kudu
/tmp/dist-test-taskAzNNjS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790284076800039-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790284076800039-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790284076800039-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790284076800039-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33183
--keytab_file=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790284076800039-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34371
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790284076800039-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790284076800039-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790284076800039-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 21:08:22.267517 6282 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 21:08:22.267678 6282 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 21:08:22.267699 6282 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 21:08:22.268767 6282 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 21:08:22.268817 6282 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 21:08:22.268844 6282 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 21:08:22.270195 6282 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34371
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790284076800039-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790284076800039-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790284076800039-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790284076800039-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33183
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6282
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790284076800039-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 19:49:06 UTC on ccycloud.quasar-vhqjoc.root.comops.site
build id 4670864
I20260924 21:08:22.270354 6282 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 21:08:22.270498 6282 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 21:08:22.275275 6287 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 21:08:22.275252 6288 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 21:08:22.275260 6290 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 21:08:22.275473 6282 server_base.cc:1034] running on GCE node
I20260924 21:08:22.275667 6282 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 21:08:22.275873 6282 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 21:08:22.277036 6282 hybrid_clock.cc:648] HybridClock initialized: now 1790284102277026 us; error 41 us; skew 500 ppm
Sep 24 21:08:22 dist-test-slave-c7tk krb5kdc[5907](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284102, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 21:08:22.279178 6282 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 21:08:22.279584 6282 webserver.cc:460] Webserver started at http://127.0.98.3:39737/ using document root <none> and password file <none>
I20260924 21:08:22.279785 6282 fs_manager.cc:359] Metadata directory not provided
I20260924 21:08:22.279845 6282 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 21:08:22.279983 6282 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 21:08:22.280833 6282 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790284076800039-392-0/minicluster-data/ts-2/data/instance:
uuid: "4fa7d620449c42ed88d0fd38edfdaf66"
format_stamp: "Formatted at 2026-09-24 21:08:22 on dist-test-slave-c7tk"
server_key: "96f634150457c71f9be04e768781eb0f"
server_key_iv: "cb47812559cc4ff3b86eadcf46b327c7"
server_key_version: "encryptionkey@0"
I20260924 21:08:22.281152 6282 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790284076800039-392-0/minicluster-data/ts-2/wal/instance:
uuid: "4fa7d620449c42ed88d0fd38edfdaf66"
format_stamp: "Formatted at 2026-09-24 21:08:22 on dist-test-slave-c7tk"
server_key: "96f634150457c71f9be04e768781eb0f"
server_key_iv: "cb47812559cc4ff3b86eadcf46b327c7"
server_key_version: "encryptionkey@0"
I20260924 21:08:22.282325 6282 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260924 21:08:22.283039 6297 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 21:08:22.283186 6282 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260924 21:08:22.283267 6282 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790284076800039-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790284076800039-392-0/minicluster-data/ts-2/wal
uuid: "4fa7d620449c42ed88d0fd38edfdaf66"
format_stamp: "Formatted at 2026-09-24 21:08:22 on dist-test-slave-c7tk"
server_key: "96f634150457c71f9be04e768781eb0f"
server_key_iv: "cb47812559cc4ff3b86eadcf46b327c7"
server_key_version: "encryptionkey@0"
I20260924 21:08:22.283334 6282 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790284076800039-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790284076800039-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790284076800039-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 21:08:22.320261 6282 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 21:08:22.322089 6282 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 21:08:22.322227 6282 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 21:08:22.322433 6282 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 21:08:22.322700 6282 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 21:08:22.322747 6282 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 21:08:22.322784 6282 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 21:08:22.322809 6282 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 21:08:22.329052 6282 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:39849
I20260924 21:08:22.329123 6410 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:39849 every 8 connection(s)
I20260924 21:08:22.329403 6282 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790284076800039-392-0/minicluster-data/ts-2/data/info.pb
Sep 24 21:08:22 dist-test-slave-c7tk krb5kdc[5907](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284102, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 21:08:22.337443 6011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:22.330127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:40287 (local address 127.0.98.62:33183)
0924 21:08:22.330204 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0924 21:08:22.330207 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 21:08:22.330355 (+ 148us) server_negotiation.cc:407] Connection header received
0924 21:08:22.331181 (+ 826us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:08:22.331182 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:08:22.331191 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:08:22.331237 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:08:22.331721 (+ 484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:22.332195 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:22.333069 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:22.333236 (+ 167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:22.333282 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:22.335225 (+ 1943us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 21:08:22.335225 (+ 0us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 21:08:22.335226 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 21:08:22.335231 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 21:08:22.336318 (+ 1087us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:08:22.336662 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:08:22.336663 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:08:22.336663 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:08:22.336723 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:08:22.336946 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:08:22.336947 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:08:22.336947 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:08:22.337057 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 21:08:22.337115 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0924 21:08:22.337318 (+ 203us) server_negotiation.cc:299] Negotiation successful
0924 21:08:22.337384 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260924 21:08:22.337546 6413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:22.330074 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33183 (local address 127.0.98.3:40287)
0924 21:08:22.330234 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0924 21:08:22.330264 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0924 21:08:22.331104 (+ 840us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 21:08:22.331284 (+ 180us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 21:08:22.331287 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 21:08:22.331499 (+ 212us) client_negotiation.cc:189] Negotiated authn=SASL
0924 21:08:22.331660 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:22.331663 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:22.332265 (+ 602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:22.332265 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:22.333004 (+ 739us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:22.333006 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:22.333299 (+ 293us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:22.333299 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:22.333400 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:22.333906 (+ 506us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 21:08:22.333913 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0924 21:08:22.335154 (+ 1241us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 21:08:22.336381 (+ 1227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 21:08:22.336386 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 21:08:22.336387 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 21:08:22.336577 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 21:08:22.336791 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 21:08:22.336791 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 21:08:22.336792 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 21:08:22.336890 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 21:08:22.337113 (+ 223us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 21:08:22.337116 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 21:08:22.337211 (+ 95us) client_negotiation.cc:769] Sending connection context
0924 21:08:22.337283 (+ 72us) client_negotiation.cc:240] Negotiation successful
0924 21:08:22.337387 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":89,"threads_started":1}
I20260924 21:08:22.338068 6411 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33183
I20260924 21:08:22.338136 6411 heartbeater.cc:464] Registering TS with master...
I20260924 21:08:22.338311 6411 heartbeater.cc:511] Master 127.0.98.62:33183 requested a full tablet report, sending...
I20260924 21:08:22.338694 5952 ts_manager.cc:194] Registered new tserver with Master: 4fa7d620449c42ed88d0fd38edfdaf66 (127.0.98.3:39849)
I20260924 21:08:22.339121 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAzNNjS/build/release/bin/kudu as pid 6282
I20260924 21:08:22.339184 5952 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:40287
I20260924 21:08:22.339219 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790284076800039-392-0/minicluster-data/ts-2/wal/instance
I20260924 21:08:22.339432 392 external_mini_cluster.cc:1428] Setting key bcdc1e3f2e7ded35b1ca645cadabc125
I20260924 21:08:22.340306 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790284076800039-392-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790284076800039-392-0/krb5kdc/test-admin.keytab.
Sep 24 21:08:22 dist-test-slave-c7tk krb5kdc[5907](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284102, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 21:08:22.354343 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260924 21:08:22.355877 6418 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 21:08:22.355088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45664 (local address 127.0.98.1:41109)
0924 21:08:22.355226 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0924 21:08:22.355230 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 21:08:22.355240 (+ 10us) server_negotiation.cc:407] Connection header received
0924 21:08:22.355267 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:08:22.355269 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:08:22.355280 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:08:22.355345 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:08:22.355798 (+ 453us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45664: BlockingRecv error: recv got EOF from 127.0.0.1:45664 (error 108)
Metrics: {"server-negotiator.queue_time_us":108,"thread_start_us":81,"threads_started":1}
W20260924 21:08:22.355986 392 init.cc:244] Error while opening credential cache '/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790284076800039-392-0/krb5kdc/krb5cc' of type FILE: Runtime error: End of credential cache reached (filename: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790284076800039-392-0/krb5kdc/krb5cc)
Sep 24 21:08:22 dist-test-slave-c7tk krb5kdc[5907](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284102, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 21:08:22.356860 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 24 21:08:22 dist-test-slave-c7tk krb5kdc[5907](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284102, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 21:08:22.357610 392 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 24 21:08:22 dist-test-slave-c7tk krb5kdc[5907](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284102, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260924 21:08:22.363866 6418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:22.358093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45674 (local address 127.0.98.1:41109)
0924 21:08:22.358122 (+ 29us) server_negotiation.cc:206] Beginning negotiation
0924 21:08:22.358123 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 21:08:22.358130 (+ 7us) server_negotiation.cc:407] Connection header received
0924 21:08:22.358176 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:08:22.358176 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:08:22.358183 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:08:22.358228 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:08:22.358581 (+ 353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:22.359257 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:22.359879 (+ 622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:22.360161 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:22.360208 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:22.361780 (+ 1572us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 21:08:22.361785 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 21:08:22.361786 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 21:08:22.361791 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 21:08:22.362996 (+ 1205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:08:22.363298 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:08:22.363300 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:08:22.363300 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:08:22.363357 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:08:22.363497 (+ 140us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:08:22.363497 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:08:22.363497 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:08:22.363594 (+ 97us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 21:08:22.363670 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0924 21:08:22.363747 (+ 77us) server_negotiation.cc:299] Negotiation successful
0924 21:08:22.363799 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14}
W20260924 21:08:22.365351 6418 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 21:08:22.364901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45690 (local address 127.0.98.1:41109)
0924 21:08:22.364943 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0924 21:08:22.364945 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 21:08:22.364975 (+ 30us) server_negotiation.cc:407] Connection header received
0924 21:08:22.365034 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:08:22.365034 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:08:22.365043 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:08:22.365086 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:08:22.365305 (+ 219us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45690: BlockingRecv error: recv got EOF from 127.0.0.1:45690 (error 108)
Metrics: {"server-negotiator.queue_time_us":26}
W20260924 21:08:22.365388 392 init.cc:244] Error while opening credential cache '/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790284076800039-392-0/krb5kdc/krb5cc' of type FILE: Runtime error: End of credential cache reached (filename: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790284076800039-392-0/krb5kdc/krb5cc)
Sep 24 21:08:22 dist-test-slave-c7tk krb5kdc[5907](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284102, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 21:08:22.366143 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 24 21:08:22 dist-test-slave-c7tk krb5kdc[5907](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284102, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 21:08:22.366814 392 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 24 21:08:22 dist-test-slave-c7tk krb5kdc[5907](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284102, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260924 21:08:22.372707 6418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:22.367179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45698 (local address 127.0.98.1:41109)
0924 21:08:22.367208 (+ 29us) server_negotiation.cc:206] Beginning negotiation
0924 21:08:22.367210 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 21:08:22.367247 (+ 37us) server_negotiation.cc:407] Connection header received
0924 21:08:22.367305 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:08:22.367306 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:08:22.367314 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:08:22.367360 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:08:22.367655 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:22.368033 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:22.368672 (+ 639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:22.368868 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:22.368947 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:22.370691 (+ 1744us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 21:08:22.370692 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 21:08:22.370692 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 21:08:22.370698 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 21:08:22.371669 (+ 971us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:08:22.371990 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:08:22.371991 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:08:22.371991 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:08:22.372043 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:08:22.372236 (+ 193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:08:22.372237 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:08:22.372238 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:08:22.372327 (+ 89us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 21:08:22.372413 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0924 21:08:22.372530 (+ 117us) server_negotiation.cc:299] Negotiation successful
0924 21:08:22.372607 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
W20260924 21:08:22.374022 6418 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 21:08:22.373464 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45712 (local address 127.0.98.1:41109)
0924 21:08:22.373492 (+ 28us) server_negotiation.cc:206] Beginning negotiation
0924 21:08:22.373493 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 21:08:22.373620 (+ 127us) server_negotiation.cc:407] Connection header received
0924 21:08:22.373669 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:08:22.373670 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:08:22.373677 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:08:22.373726 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:08:22.373984 (+ 258us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45712: BlockingRecv error: recv got EOF from 127.0.0.1:45712 (error 108)
Metrics: {"server-negotiator.queue_time_us":13}
W20260924 21:08:22.374101 392 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790284076800039-392-0/krb5kdc/krb5cc' of type FILE
Sep 24 21:08:22 dist-test-slave-c7tk krb5kdc[5907](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284102, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 21:08:22.374958 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 24 21:08:22 dist-test-slave-c7tk krb5kdc[5907](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284102, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 21:08:22.375730 392 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 24 21:08:22 dist-test-slave-c7tk krb5kdc[5907](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284102, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260924 21:08:22.381570 6418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:22.376142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45718 (local address 127.0.98.1:41109)
0924 21:08:22.376173 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0924 21:08:22.376175 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 21:08:22.376217 (+ 42us) server_negotiation.cc:407] Connection header received
0924 21:08:22.376275 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:08:22.376276 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:08:22.376283 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:08:22.376335 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:08:22.376676 (+ 341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:22.377051 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:22.377646 (+ 595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:22.377891 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:22.377959 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:22.379682 (+ 1723us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 21:08:22.379683 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 21:08:22.379683 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 21:08:22.379688 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 21:08:22.380804 (+ 1116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:08:22.381070 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:08:22.381071 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:08:22.381071 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:08:22.381111 (+ 40us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:08:22.381233 (+ 122us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:08:22.381234 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:08:22.381234 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:08:22.381293 (+ 59us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 21:08:22.381353 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0924 21:08:22.381418 (+ 65us) server_negotiation.cc:299] Negotiation successful
0924 21:08:22.381460 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260924 21:08:22.382077 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAzNNjS/build/release/bin/kudu with pid 6014
I20260924 21:08:22.386276 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAzNNjS/build/release/bin/kudu with pid 6148
I20260924 21:08:22.390492 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAzNNjS/build/release/bin/kudu with pid 6282
I20260924 21:08:22.394936 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAzNNjS/build/release/bin/kudu with pid 5920
2026-09-24T21:08:22Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (1683 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790284076800039-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 21:08:22 dist-test-slave-c7tk krb5kdc[6431](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 21:08:22 dist-test-slave-c7tk krb5kdc[6431](info): set up 2 sockets
Sep 24 21:08:22 dist-test-slave-c7tk krb5kdc[6431](info): commencing operation
krb5kdc: starting...
W20260924 21:08:23.341653 392 mini_kdc.cc:121] Time spent starting KDC: real 0.927s user 0.000s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790284076800039-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790284076800039-392-0/krb5kdc/test-user.keytab.
Sep 24 21:08:23 dist-test-slave-c7tk krb5kdc[6431](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284103, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T21:08:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T21:08:23Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790284076800039-392-0/krb5kdc/oryx_127.0.98.62.keytab.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790284076800039-392-0/krb5kdc/oryx_127.0.98.62.keytab.
I20260924 21:08:23.442431 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAzNNjS/build/release/bin/kudu
/tmp/dist-test-taskAzNNjS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790284076800039-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790284076800039-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790284076800039-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790284076800039-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44697
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790284076800039-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37913
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44697
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790284076800039-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790284076800039-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790284076800039-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 21:08:23.513712 6444 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 21:08:23.513936 6444 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 21:08:23.513959 6444 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 21:08:23.515035 6444 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 21:08:23.515072 6444 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 21:08:23.515085 6444 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 21:08:23.515097 6444 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 21:08:23.515108 6444 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 21:08:23.516402 6444 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37913
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790284076800039-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790284076800039-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44697
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790284076800039-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44697
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790284076800039-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6444
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790284076800039-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 19:49:06 UTC on ccycloud.quasar-vhqjoc.root.comops.site
build id 4670864
I20260924 21:08:23.516551 6444 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 21:08:23.516680 6444 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 21:08:23.520889 6452 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 21:08:23.520927 6449 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 21:08:23.520972 6450 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 21:08:23.521016 6444 server_base.cc:1034] running on GCE node
I20260924 21:08:23.521411 6444 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 21:08:23.521646 6444 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 21:08:23.522791 6444 hybrid_clock.cc:648] HybridClock initialized: now 1790284103522784 us; error 23 us; skew 500 ppm
Sep 24 21:08:23 dist-test-slave-c7tk krb5kdc[6431](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284103, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 21:08:23.524833 6444 init.cc:377] Logged in from keytab as oryx/127.0.98.62@KRBTEST.COM (short username oryx)
I20260924 21:08:23.525213 6444 webserver.cc:460] Webserver started at http://127.0.98.62:45385/ using document root <none> and password file <none>
I20260924 21:08:23.525362 6444 fs_manager.cc:359] Metadata directory not provided
I20260924 21:08:23.525398 6444 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 21:08:23.525472 6444 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 21:08:23.526367 6444 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790284076800039-392-0/minicluster-data/master-0/data/instance:
uuid: "0f53f4b63d8e4da0905f8db79d1fc7db"
format_stamp: "Formatted at 2026-09-24 21:08:23 on dist-test-slave-c7tk"
server_key: "bf54c801dfed6fa0714ec864ad2e4b40"
server_key_iv: "f9857b5c722c709803fe0d2a8db2e65a"
server_key_version: "encryptionkey@0"
I20260924 21:08:23.526660 6444 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790284076800039-392-0/minicluster-data/master-0/wal/instance:
uuid: "0f53f4b63d8e4da0905f8db79d1fc7db"
format_stamp: "Formatted at 2026-09-24 21:08:23 on dist-test-slave-c7tk"
server_key: "bf54c801dfed6fa0714ec864ad2e4b40"
server_key_iv: "f9857b5c722c709803fe0d2a8db2e65a"
server_key_version: "encryptionkey@0"
I20260924 21:08:23.527796 6444 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 21:08:23.528363 6459 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 21:08:23.528566 6444 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 21:08:23.528651 6444 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790284076800039-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790284076800039-392-0/minicluster-data/master-0/wal
uuid: "0f53f4b63d8e4da0905f8db79d1fc7db"
format_stamp: "Formatted at 2026-09-24 21:08:23 on dist-test-slave-c7tk"
server_key: "bf54c801dfed6fa0714ec864ad2e4b40"
server_key_iv: "f9857b5c722c709803fe0d2a8db2e65a"
server_key_version: "encryptionkey@0"
I20260924 21:08:23.528719 6444 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790284076800039-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790284076800039-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790284076800039-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 21:08:23.546941 6444 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 21:08:23.548318 6444 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 21:08:23.548465 6444 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 21:08:23.552228 6444 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44697
I20260924 21:08:23.552255 6511 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44697 every 8 connection(s)
I20260924 21:08:23.552542 6444 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790284076800039-392-0/minicluster-data/master-0/data/info.pb
I20260924 21:08:23.553313 6512 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 21:08:23.555267 6512 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0f53f4b63d8e4da0905f8db79d1fc7db: Bootstrap starting.
I20260924 21:08:23.555766 6512 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0f53f4b63d8e4da0905f8db79d1fc7db: Neither blocks nor log segments found. Creating new log.
I20260924 21:08:23.556041 6512 log.cc:824] T 00000000000000000000000000000000 P 0f53f4b63d8e4da0905f8db79d1fc7db: Log is configured to *not* fsync() on all Append() calls
I20260924 21:08:23.556047 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAzNNjS/build/release/bin/kudu as pid 6444
I20260924 21:08:23.556142 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790284076800039-392-0/minicluster-data/master-0/wal/instance
I20260924 21:08:23.556340 392 external_mini_cluster.cc:1428] Setting key 957ee22bf5c7458a5b64e24e8704616a
I20260924 21:08:23.556699 6512 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0f53f4b63d8e4da0905f8db79d1fc7db: No bootstrap required, opened a new log
I20260924 21:08:23.558267 6512 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0f53f4b63d8e4da0905f8db79d1fc7db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f53f4b63d8e4da0905f8db79d1fc7db" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44697 } }
I20260924 21:08:23.558410 6512 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0f53f4b63d8e4da0905f8db79d1fc7db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 21:08:23.558451 6512 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0f53f4b63d8e4da0905f8db79d1fc7db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f53f4b63d8e4da0905f8db79d1fc7db, State: Initialized, Role: FOLLOWER
I20260924 21:08:23.558616 6512 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0f53f4b63d8e4da0905f8db79d1fc7db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f53f4b63d8e4da0905f8db79d1fc7db" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44697 } }
I20260924 21:08:23.558698 6512 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0f53f4b63d8e4da0905f8db79d1fc7db [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 21:08:23.558732 6512 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0f53f4b63d8e4da0905f8db79d1fc7db [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 21:08:23.558780 6512 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0f53f4b63d8e4da0905f8db79d1fc7db [term 0 FOLLOWER]: Advancing to term 1
I20260924 21:08:23.559695 6512 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0f53f4b63d8e4da0905f8db79d1fc7db [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f53f4b63d8e4da0905f8db79d1fc7db" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44697 } }
I20260924 21:08:23.559784 6512 leader_election.cc:302] T 00000000000000000000000000000000 P 0f53f4b63d8e4da0905f8db79d1fc7db [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0f53f4b63d8e4da0905f8db79d1fc7db; no voters:
I20260924 21:08:23.559939 6512 leader_election.cc:288] T 00000000000000000000000000000000 P 0f53f4b63d8e4da0905f8db79d1fc7db [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 21:08:23.559958 6517 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0f53f4b63d8e4da0905f8db79d1fc7db [term 1 FOLLOWER]: Leader election won for term 1
I20260924 21:08:23.560068 6517 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0f53f4b63d8e4da0905f8db79d1fc7db [term 1 LEADER]: Becoming Leader. State: Replica: 0f53f4b63d8e4da0905f8db79d1fc7db, State: Running, Role: LEADER
I20260924 21:08:23.560137 6517 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0f53f4b63d8e4da0905f8db79d1fc7db [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f53f4b63d8e4da0905f8db79d1fc7db" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44697 } }
I20260924 21:08:23.560158 6512 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0f53f4b63d8e4da0905f8db79d1fc7db [sys.catalog]: configured and running, proceeding with master startup.
I20260924 21:08:23.561874 6518 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0f53f4b63d8e4da0905f8db79d1fc7db [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0f53f4b63d8e4da0905f8db79d1fc7db" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f53f4b63d8e4da0905f8db79d1fc7db" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44697 } } }
I20260924 21:08:23.561879 6519 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0f53f4b63d8e4da0905f8db79d1fc7db [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0f53f4b63d8e4da0905f8db79d1fc7db. Latest consensus state: current_term: 1 leader_uuid: "0f53f4b63d8e4da0905f8db79d1fc7db" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f53f4b63d8e4da0905f8db79d1fc7db" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44697 } } }
I20260924 21:08:23.562258 6518 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0f53f4b63d8e4da0905f8db79d1fc7db [sys.catalog]: This master's current role is: LEADER
I20260924 21:08:23.562268 6519 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0f53f4b63d8e4da0905f8db79d1fc7db [sys.catalog]: This master's current role is: LEADER
W20260924 21:08:23.562824 6531 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0f53f4b63d8e4da0905f8db79d1fc7db: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 21:08:23.562906 6531 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 21:08:23.562973 6533 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 21:08:23.563129 6533 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 21:08:23.564373 6533 catalog_manager.cc:1347] Generated new cluster ID: e2489afc08c64420b12bd07cdab5f107
Sep 24 21:08:23 dist-test-slave-c7tk krb5kdc[6431](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284103, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260924 21:08:23.564419 6533 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 21:08:23.568419 6516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:23.556834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33594 (local address 127.0.98.62:44697)
0924 21:08:23.557004 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0924 21:08:23.557012 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 21:08:23.557043 (+ 31us) server_negotiation.cc:407] Connection header received
0924 21:08:23.558253 (+ 1210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:08:23.558258 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:08:23.558298 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:08:23.558401 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:08:23.559213 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:23.560533 (+ 1320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:23.561321 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:23.562062 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:23.562190 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:23.564954 (+ 2764us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 21:08:23.564958 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 21:08:23.564958 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 21:08:23.564968 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0924 21:08:23.566896 (+ 1928us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:08:23.567297 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:08:23.567299 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:08:23.567300 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:08:23.567416 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:08:23.567622 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:08:23.567623 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:08:23.567623 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:08:23.567877 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 21:08:23.567999 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0924 21:08:23.568124 (+ 125us) server_negotiation.cc:299] Negotiation successful
0924 21:08:23.568238 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":94,"threads_started":1}
I20260924 21:08:23.587308 6533 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 21:08:23.587793 6533 catalog_manager.cc:1504] Loading token signing keys...
I20260924 21:08:23.594372 6533 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0f53f4b63d8e4da0905f8db79d1fc7db: Generated new TSK 0
I20260924 21:08:23.594525 6533 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790284076800039-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790284076800039-392-0/krb5kdc/oryx_127.0.98.1.keytab.
I20260924 21:08:23.626219 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAzNNjS/build/release/bin/kudu
/tmp/dist-test-taskAzNNjS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790284076800039-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790284076800039-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790284076800039-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790284076800039-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44697
--keytab_file=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790284076800039-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37913
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790284076800039-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790284076800039-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790284076800039-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 21:08:23.699045 6538 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 21:08:23.699224 6538 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 21:08:23.699246 6538 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 21:08:23.700400 6538 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 21:08:23.700440 6538 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 21:08:23.700466 6538 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 21:08:23.701913 6538 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37913
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790284076800039-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790284076800039-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790284076800039-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790284076800039-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44697
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6538
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790284076800039-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 19:49:06 UTC on ccycloud.quasar-vhqjoc.root.comops.site
build id 4670864
I20260924 21:08:23.702135 6538 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 21:08:23.702343 6538 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 21:08:23.707002 6546 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 21:08:23.707091 6543 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 21:08:23.707015 6544 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 21:08:23.707193 6538 server_base.cc:1034] running on GCE node
I20260924 21:08:23.707350 6538 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 21:08:23.707510 6538 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 21:08:23.708662 6538 hybrid_clock.cc:648] HybridClock initialized: now 1790284103708649 us; error 23 us; skew 500 ppm
Sep 24 21:08:23 dist-test-slave-c7tk krb5kdc[6431](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284103, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 21:08:23.710917 6538 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260924 21:08:23.711272 6538 webserver.cc:460] Webserver started at http://127.0.98.1:39407/ using document root <none> and password file <none>
I20260924 21:08:23.711480 6538 fs_manager.cc:359] Metadata directory not provided
I20260924 21:08:23.711539 6538 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 21:08:23.711683 6538 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 21:08:23.712504 6538 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790284076800039-392-0/minicluster-data/ts-0/data/instance:
uuid: "1111a5950630489ba30bd8b925f7a15b"
format_stamp: "Formatted at 2026-09-24 21:08:23 on dist-test-slave-c7tk"
server_key: "7b301a44a3bb1dcac9b0c435adc707b8"
server_key_iv: "548b23a9211ad056ee847c20876171f9"
server_key_version: "encryptionkey@0"
I20260924 21:08:23.712818 6538 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790284076800039-392-0/minicluster-data/ts-0/wal/instance:
uuid: "1111a5950630489ba30bd8b925f7a15b"
format_stamp: "Formatted at 2026-09-24 21:08:23 on dist-test-slave-c7tk"
server_key: "7b301a44a3bb1dcac9b0c435adc707b8"
server_key_iv: "548b23a9211ad056ee847c20876171f9"
server_key_version: "encryptionkey@0"
I20260924 21:08:23.714159 6538 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 21:08:23.715075 6553 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 21:08:23.715230 6538 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 21:08:23.715320 6538 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790284076800039-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790284076800039-392-0/minicluster-data/ts-0/wal
uuid: "1111a5950630489ba30bd8b925f7a15b"
format_stamp: "Formatted at 2026-09-24 21:08:23 on dist-test-slave-c7tk"
server_key: "7b301a44a3bb1dcac9b0c435adc707b8"
server_key_iv: "548b23a9211ad056ee847c20876171f9"
server_key_version: "encryptionkey@0"
I20260924 21:08:23.715394 6538 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790284076800039-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790284076800039-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790284076800039-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 21:08:23.724007 6538 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 21:08:23.725556 6538 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 21:08:23.725690 6538 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 21:08:23.725915 6538 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 21:08:23.726182 6538 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 21:08:23.726229 6538 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 21:08:23.726263 6538 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 21:08:23.726289 6538 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 21:08:23.731376 6538 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:41965
I20260924 21:08:23.731443 6666 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:41965 every 8 connection(s)
I20260924 21:08:23.731673 6538 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790284076800039-392-0/minicluster-data/ts-0/data/info.pb
Sep 24 21:08:23 dist-test-slave-c7tk krb5kdc[6431](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284103, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260924 21:08:23.739341 6516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:23.732293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:60827 (local address 127.0.98.62:44697)
0924 21:08:23.732328 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0924 21:08:23.732330 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 21:08:23.732513 (+ 183us) server_negotiation.cc:407] Connection header received
0924 21:08:23.732926 (+ 413us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:08:23.732927 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:08:23.732940 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:08:23.732988 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:08:23.733828 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:23.734322 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:23.735216 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:23.735404 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:23.735492 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:23.737227 (+ 1735us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 21:08:23.737228 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 21:08:23.737228 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 21:08:23.737236 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 21:08:23.738291 (+ 1055us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:08:23.738636 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:08:23.738636 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:08:23.738637 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:08:23.738681 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:08:23.738880 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:08:23.738881 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:08:23.738881 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:08:23.738969 (+ 88us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 21:08:23.739016 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0924 21:08:23.739220 (+ 204us) server_negotiation.cc:299] Negotiation successful
0924 21:08:23.739281 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260924 21:08:23.739508 6669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:23.732265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44697 (local address 127.0.98.1:60827)
0924 21:08:23.732420 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0924 21:08:23.732446 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0924 21:08:23.732826 (+ 380us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 21:08:23.733043 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 21:08:23.733047 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 21:08:23.733520 (+ 473us) client_negotiation.cc:189] Negotiated authn=SASL
0924 21:08:23.733709 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:23.733712 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:23.734428 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:23.734429 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:23.735127 (+ 698us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:23.735129 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:23.735497 (+ 368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:23.735498 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:23.735602 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:23.736003 (+ 401us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 21:08:23.736011 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 21:08:23.737143 (+ 1132us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 21:08:23.738353 (+ 1210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 21:08:23.738358 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 21:08:23.738359 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 21:08:23.738573 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 21:08:23.738737 (+ 164us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 21:08:23.738738 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 21:08:23.738738 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 21:08:23.738822 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 21:08:23.739020 (+ 198us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 21:08:23.739024 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 21:08:23.739120 (+ 96us) client_negotiation.cc:769] Sending connection context
0924 21:08:23.739197 (+ 77us) client_negotiation.cc:240] Negotiation successful
0924 21:08:23.739299 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":73,"threads_started":1}
I20260924 21:08:23.739970 6667 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44697
I20260924 21:08:23.739977 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAzNNjS/build/release/bin/kudu as pid 6538
I20260924 21:08:23.740064 6667 heartbeater.cc:464] Registering TS with master...
I20260924 21:08:23.740072 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790284076800039-392-0/minicluster-data/ts-0/wal/instance
I20260924 21:08:23.740206 6667 heartbeater.cc:511] Master 127.0.98.62:44697 requested a full tablet report, sending...
I20260924 21:08:23.740242 392 external_mini_cluster.cc:1428] Setting key 511a306e899137e0e39aee1f87ed2d92
I20260924 21:08:23.740648 6476 ts_manager.cc:194] Registered new tserver with Master: 1111a5950630489ba30bd8b925f7a15b (127.0.98.1:41965)
I20260924 21:08:23.741418 6476 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:60827
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790284076800039-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790284076800039-392-0/krb5kdc/oryx_127.0.98.2.keytab.
I20260924 21:08:23.760617 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAzNNjS/build/release/bin/kudu
/tmp/dist-test-taskAzNNjS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790284076800039-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790284076800039-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790284076800039-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790284076800039-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44697
--keytab_file=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790284076800039-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37913
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790284076800039-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790284076800039-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790284076800039-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 21:08:23.831841 6672 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 21:08:23.832031 6672 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 21:08:23.832065 6672 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 21:08:23.833148 6672 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 21:08:23.833204 6672 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 21:08:23.833245 6672 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 21:08:23.834532 6672 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37913
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790284076800039-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790284076800039-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790284076800039-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790284076800039-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44697
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6672
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790284076800039-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 19:49:06 UTC on ccycloud.quasar-vhqjoc.root.comops.site
build id 4670864
I20260924 21:08:23.834769 6672 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 21:08:23.834968 6672 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 21:08:23.839329 6680 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 21:08:23.839422 6672 server_base.cc:1034] running on GCE node
W20260924 21:08:23.839488 6678 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 21:08:23.839351 6677 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 21:08:23.839769 6672 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 21:08:23.839977 6672 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 21:08:23.841115 6672 hybrid_clock.cc:648] HybridClock initialized: now 1790284103841094 us; error 38 us; skew 500 ppm
Sep 24 21:08:23 dist-test-slave-c7tk krb5kdc[6431](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284103, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 21:08:23.843120 6672 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260924 21:08:23.843405 6672 webserver.cc:460] Webserver started at http://127.0.98.2:37657/ using document root <none> and password file <none>
I20260924 21:08:23.843570 6672 fs_manager.cc:359] Metadata directory not provided
I20260924 21:08:23.843606 6672 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 21:08:23.843680 6672 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 21:08:23.844532 6672 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790284076800039-392-0/minicluster-data/ts-1/data/instance:
uuid: "80a0113d8614402db4980fda67be9b4f"
format_stamp: "Formatted at 2026-09-24 21:08:23 on dist-test-slave-c7tk"
server_key: "4cead28aafe68f4b0c1e5bb4ef186ccc"
server_key_iv: "5cfc7a0724ce40c647813a9d18503e34"
server_key_version: "encryptionkey@0"
I20260924 21:08:23.844781 6672 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790284076800039-392-0/minicluster-data/ts-1/wal/instance:
uuid: "80a0113d8614402db4980fda67be9b4f"
format_stamp: "Formatted at 2026-09-24 21:08:23 on dist-test-slave-c7tk"
server_key: "4cead28aafe68f4b0c1e5bb4ef186ccc"
server_key_iv: "5cfc7a0724ce40c647813a9d18503e34"
server_key_version: "encryptionkey@0"
I20260924 21:08:23.845765 6672 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 21:08:23.846522 6687 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 21:08:23.846700 6672 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 21:08:23.846766 6672 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790284076800039-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790284076800039-392-0/minicluster-data/ts-1/wal
uuid: "80a0113d8614402db4980fda67be9b4f"
format_stamp: "Formatted at 2026-09-24 21:08:23 on dist-test-slave-c7tk"
server_key: "4cead28aafe68f4b0c1e5bb4ef186ccc"
server_key_iv: "5cfc7a0724ce40c647813a9d18503e34"
server_key_version: "encryptionkey@0"
I20260924 21:08:23.846851 6672 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790284076800039-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790284076800039-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790284076800039-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 21:08:23.881121 6672 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 21:08:23.882793 6672 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 21:08:23.882948 6672 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 21:08:23.883152 6672 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 21:08:23.883445 6672 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 21:08:23.883499 6672 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 21:08:23.883539 6672 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 21:08:23.883566 6672 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 21:08:23.888773 6672 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:46665
I20260924 21:08:23.888864 6800 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:46665 every 8 connection(s)
I20260924 21:08:23.889140 6672 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790284076800039-392-0/minicluster-data/ts-1/data/info.pb
Sep 24 21:08:23 dist-test-slave-c7tk krb5kdc[6431](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284103, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260924 21:08:23.894600 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAzNNjS/build/release/bin/kudu as pid 6672
I20260924 21:08:23.894671 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790284076800039-392-0/minicluster-data/ts-1/wal/instance
I20260924 21:08:23.894927 392 external_mini_cluster.cc:1428] Setting key 66c0f8a085cca5612634719ec53246e6
I20260924 21:08:23.897642 6516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:23.889849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:43963 (local address 127.0.98.62:44697)
0924 21:08:23.889902 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0924 21:08:23.889905 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 21:08:23.890070 (+ 165us) server_negotiation.cc:407] Connection header received
0924 21:08:23.890528 (+ 458us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:08:23.890529 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:08:23.890541 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:08:23.890590 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:08:23.891346 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:23.891782 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:23.892685 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:23.892885 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:23.892962 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:23.894854 (+ 1892us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 21:08:23.894856 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 21:08:23.894856 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 21:08:23.894863 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 21:08:23.896407 (+ 1544us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:08:23.896773 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:08:23.896774 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:08:23.896775 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:08:23.896830 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:08:23.897101 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:08:23.897102 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:08:23.897102 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:08:23.897194 (+ 92us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 21:08:23.897240 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0924 21:08:23.897532 (+ 292us) server_negotiation.cc:299] Negotiation successful
0924 21:08:23.897588 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260924 21:08:23.897919 6803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:23.889780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44697 (local address 127.0.98.2:43963)
0924 21:08:23.889962 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0924 21:08:23.890000 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0924 21:08:23.890430 (+ 430us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 21:08:23.890648 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 21:08:23.890652 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 21:08:23.891111 (+ 459us) client_negotiation.cc:189] Negotiated authn=SASL
0924 21:08:23.891261 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:23.891265 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:23.891862 (+ 597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:23.891863 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:23.892589 (+ 726us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:23.892591 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:23.892968 (+ 377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:23.892969 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:23.893082 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:23.893438 (+ 356us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 21:08:23.893444 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0924 21:08:23.894757 (+ 1313us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 21:08:23.896475 (+ 1718us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 21:08:23.896480 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 21:08:23.896481 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 21:08:23.896700 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 21:08:23.896900 (+ 200us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 21:08:23.896901 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 21:08:23.896901 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 21:08:23.897041 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 21:08:23.897252 (+ 211us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 21:08:23.897256 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 21:08:23.897412 (+ 156us) client_negotiation.cc:769] Sending connection context
0924 21:08:23.897516 (+ 104us) client_negotiation.cc:240] Negotiation successful
0924 21:08:23.897662 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":101,"threads_started":1}
I20260924 21:08:23.898281 6801 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44697
I20260924 21:08:23.898367 6801 heartbeater.cc:464] Registering TS with master...
I20260924 21:08:23.898533 6801 heartbeater.cc:511] Master 127.0.98.62:44697 requested a full tablet report, sending...
I20260924 21:08:23.898882 6476 ts_manager.cc:194] Registered new tserver with Master: 80a0113d8614402db4980fda67be9b4f (127.0.98.2:46665)
I20260924 21:08:23.899281 6476 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:43963
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790284076800039-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790284076800039-392-0/krb5kdc/oryx_127.0.98.3.keytab.
I20260924 21:08:23.916157 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAzNNjS/build/release/bin/kudu
/tmp/dist-test-taskAzNNjS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790284076800039-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790284076800039-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790284076800039-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790284076800039-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44697
--keytab_file=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790284076800039-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37913
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790284076800039-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790284076800039-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790284076800039-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 21:08:23.986070 6806 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 21:08:23.986249 6806 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 21:08:23.986279 6806 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 21:08:23.987277 6806 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 21:08:23.987334 6806 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 21:08:23.987372 6806 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 21:08:23.988605 6806 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37913
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790284076800039-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790284076800039-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790284076800039-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790284076800039-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44697
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6806
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790284076800039-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 19:49:06 UTC on ccycloud.quasar-vhqjoc.root.comops.site
build id 4670864
I20260924 21:08:23.988842 6806 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 21:08:23.989068 6806 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 21:08:23.993403 6811 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 21:08:23.993414 6812 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 21:08:23.993414 6814 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 21:08:23.993665 6806 server_base.cc:1034] running on GCE node
I20260924 21:08:23.993888 6806 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 21:08:23.994128 6806 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 21:08:23.995280 6806 hybrid_clock.cc:648] HybridClock initialized: now 1790284103995275 us; error 33 us; skew 500 ppm
Sep 24 21:08:23 dist-test-slave-c7tk krb5kdc[6431](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284103, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 21:08:23.997524 6806 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260924 21:08:23.997920 6806 webserver.cc:460] Webserver started at http://127.0.98.3:38997/ using document root <none> and password file <none>
I20260924 21:08:23.998109 6806 fs_manager.cc:359] Metadata directory not provided
I20260924 21:08:23.998165 6806 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 21:08:23.998273 6806 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 21:08:23.999174 6806 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790284076800039-392-0/minicluster-data/ts-2/data/instance:
uuid: "54766fc07b09440682225ff08c6814db"
format_stamp: "Formatted at 2026-09-24 21:08:23 on dist-test-slave-c7tk"
server_key: "d52417912803c1748e764c24c07463ab"
server_key_iv: "d05996b569d88ed04b52c0e475301a0a"
server_key_version: "encryptionkey@0"
I20260924 21:08:23.999526 6806 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790284076800039-392-0/minicluster-data/ts-2/wal/instance:
uuid: "54766fc07b09440682225ff08c6814db"
format_stamp: "Formatted at 2026-09-24 21:08:23 on dist-test-slave-c7tk"
server_key: "d52417912803c1748e764c24c07463ab"
server_key_iv: "d05996b569d88ed04b52c0e475301a0a"
server_key_version: "encryptionkey@0"
I20260924 21:08:24.000627 6806 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 21:08:24.001169 6821 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 21:08:24.001296 6806 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 21:08:24.001389 6806 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790284076800039-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790284076800039-392-0/minicluster-data/ts-2/wal
uuid: "54766fc07b09440682225ff08c6814db"
format_stamp: "Formatted at 2026-09-24 21:08:23 on dist-test-slave-c7tk"
server_key: "d52417912803c1748e764c24c07463ab"
server_key_iv: "d05996b569d88ed04b52c0e475301a0a"
server_key_version: "encryptionkey@0"
I20260924 21:08:24.001459 6806 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790284076800039-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790284076800039-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790284076800039-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 21:08:24.014922 6806 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 21:08:24.016957 6806 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 21:08:24.017108 6806 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 21:08:24.017313 6806 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 21:08:24.017572 6806 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 21:08:24.017639 6806 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 21:08:24.017675 6806 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 21:08:24.017704 6806 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 21:08:24.023144 6806 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:37191
I20260924 21:08:24.023175 6934 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:37191 every 8 connection(s)
I20260924 21:08:24.023422 6806 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790284076800039-392-0/minicluster-data/ts-2/data/info.pb
Sep 24 21:08:24 dist-test-slave-c7tk krb5kdc[6431](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284103, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260924 21:08:24.029865 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAzNNjS/build/release/bin/kudu as pid 6806
I20260924 21:08:24.029969 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790284076800039-392-0/minicluster-data/ts-2/wal/instance
I20260924 21:08:24.030124 392 external_mini_cluster.cc:1428] Setting key ff0e3dbb0229eb5ea45c660eea5e4981
I20260924 21:08:24.031505 6516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:24.024053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:51657 (local address 127.0.98.62:44697)
0924 21:08:24.024118 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0924 21:08:24.024120 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 21:08:24.024268 (+ 148us) server_negotiation.cc:407] Connection header received
0924 21:08:24.024642 (+ 374us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:08:24.024643 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:08:24.024652 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:08:24.024699 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:08:24.025402 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:24.025851 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:24.026676 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:24.026834 (+ 158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:24.026879 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:24.028743 (+ 1864us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 21:08:24.028744 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 21:08:24.028744 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 21:08:24.028750 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 21:08:24.030147 (+ 1397us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:08:24.030533 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:08:24.030534 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:08:24.030534 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:08:24.030587 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:08:24.030994 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:08:24.030995 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:08:24.030995 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:08:24.031082 (+ 87us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 21:08:24.031128 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0924 21:08:24.031387 (+ 259us) server_negotiation.cc:299] Negotiation successful
0924 21:08:24.031448 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260924 21:08:24.031630 6937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:24.024011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44697 (local address 127.0.98.3:51657)
0924 21:08:24.024175 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0924 21:08:24.024203 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0924 21:08:24.024572 (+ 369us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 21:08:24.024741 (+ 169us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 21:08:24.024743 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 21:08:24.025197 (+ 454us) client_negotiation.cc:189] Negotiated authn=SASL
0924 21:08:24.025343 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:24.025346 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:24.025912 (+ 566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:24.025912 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:24.026592 (+ 680us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:24.026593 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:24.026896 (+ 303us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:24.026897 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:24.026979 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:24.027440 (+ 461us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 21:08:24.027447 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0924 21:08:24.028662 (+ 1215us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 21:08:24.030232 (+ 1570us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 21:08:24.030241 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 21:08:24.030243 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 21:08:24.030470 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 21:08:24.030809 (+ 339us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 21:08:24.030810 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 21:08:24.030811 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 21:08:24.030944 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 21:08:24.031122 (+ 178us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 21:08:24.031124 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 21:08:24.031277 (+ 153us) client_negotiation.cc:769] Sending connection context
0924 21:08:24.031357 (+ 80us) client_negotiation.cc:240] Negotiation successful
0924 21:08:24.031461 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":96,"threads_started":1}
I20260924 21:08:24.032094 6935 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44697
I20260924 21:08:24.032188 6935 heartbeater.cc:464] Registering TS with master...
I20260924 21:08:24.032353 6935 heartbeater.cc:511] Master 127.0.98.62:44697 requested a full tablet report, sending...
I20260924 21:08:24.032681 6476 ts_manager.cc:194] Registered new tserver with Master: 54766fc07b09440682225ff08c6814db (127.0.98.3:37191)
I20260924 21:08:24.032998 6476 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:51657
I20260924 21:08:24.041589 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790284076800039-392-0/krb5kdc/oryx_127.0.98.61.keytab.
Entry for principal oryx/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790284076800039-392-0/krb5kdc/oryx_127.0.98.61.keytab.
I20260924 21:08:24.064808 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAzNNjS/build/release/bin/kudu
/tmp/dist-test-taskAzNNjS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790284076800039-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790284076800039-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790284076800039-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790284076800039-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:34067
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790284076800039-392-0/krb5kdc/oryx_127.0.98.61.keytab
--principal=oryx/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37913
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44697,127.0.98.61:34067
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790284076800039-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790284076800039-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790284076800039-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 21:08:24.135011 6940 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 21:08:24.135161 6940 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 21:08:24.135179 6940 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 21:08:24.136135 6940 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 21:08:24.136171 6940 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 21:08:24.136183 6940 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 21:08:24.136226 6940 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 21:08:24.136245 6940 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 21:08:24.137492 6940 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37913
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790284076800039-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790284076800039-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44697,127.0.98.61:34067
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790284076800039-392-0/krb5kdc/oryx_127.0.98.61.keytab
--principal=oryx/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:34067
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790284076800039-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6940
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790284076800039-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 19:49:06 UTC on ccycloud.quasar-vhqjoc.root.comops.site
build id 4670864
W20260924 21:08:24.137567 6940 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.0.98.62:44697,127.0.98.61:34067'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260924 21:08:24.137691 6940 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 21:08:24.137884 6940 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 21:08:24.142071 6940 server_base.cc:1034] running on GCE node
W20260924 21:08:24.142072 6946 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 21:08:24.142045 6945 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 21:08:24.142030 6948 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 21:08:24.142513 6940 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 21:08:24.142746 6940 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 21:08:24.143877 6940 hybrid_clock.cc:648] HybridClock initialized: now 1790284104143864 us; error 26 us; skew 500 ppm
Sep 24 21:08:24 dist-test-slave-c7tk krb5kdc[6431](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284104, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 21:08:24.145857 6940 init.cc:377] Logged in from keytab as oryx/127.0.98.61@KRBTEST.COM (short username oryx)
I20260924 21:08:24.146239 6940 webserver.cc:460] Webserver started at http://127.0.98.61:37251/ using document root <none> and password file <none>
I20260924 21:08:24.146430 6940 fs_manager.cc:359] Metadata directory not provided
I20260924 21:08:24.146484 6940 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 21:08:24.146590 6940 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 21:08:24.147390 6940 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790284076800039-392-0/minicluster-data/master-1/data/instance:
uuid: "a5110fb5b32e4973948078ef72376369"
format_stamp: "Formatted at 2026-09-24 21:08:24 on dist-test-slave-c7tk"
server_key: "f28599b3a36446a50e9c15952a945a86"
server_key_iv: "b41fee4f49a573c746eb3406870b7e7f"
server_key_version: "encryptionkey@0"
I20260924 21:08:24.147686 6940 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790284076800039-392-0/minicluster-data/master-1/wal/instance:
uuid: "a5110fb5b32e4973948078ef72376369"
format_stamp: "Formatted at 2026-09-24 21:08:24 on dist-test-slave-c7tk"
server_key: "f28599b3a36446a50e9c15952a945a86"
server_key_iv: "b41fee4f49a573c746eb3406870b7e7f"
server_key_version: "encryptionkey@0"
I20260924 21:08:24.148772 6940 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 21:08:24.149487 6955 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 21:08:24.149647 6940 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 21:08:24.149758 6940 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790284076800039-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790284076800039-392-0/minicluster-data/master-1/wal
uuid: "a5110fb5b32e4973948078ef72376369"
format_stamp: "Formatted at 2026-09-24 21:08:24 on dist-test-slave-c7tk"
server_key: "f28599b3a36446a50e9c15952a945a86"
server_key_iv: "b41fee4f49a573c746eb3406870b7e7f"
server_key_version: "encryptionkey@0"
I20260924 21:08:24.149863 6940 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790284076800039-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790284076800039-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790284076800039-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 21:08:24.163090 6940 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 21:08:24.164608 6940 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 21:08:24.164738 6940 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 21:08:24.168370 6940 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:34067
I20260924 21:08:24.168406 7007 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:34067 every 8 connection(s)
I20260924 21:08:24.168737 6940 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790284076800039-392-0/minicluster-data/master-1/data/info.pb
I20260924 21:08:24.169570 7008 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 21:08:24.170820 7008 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44697 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 21:08:24 dist-test-slave-c7tk krb5kdc[6431](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284104, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260924 21:08:24.177539 6516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:24.171350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33596 (local address 127.0.98.62:44697)
0924 21:08:24.171390 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0924 21:08:24.171391 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 21:08:24.171579 (+ 188us) server_negotiation.cc:407] Connection header received
0924 21:08:24.171962 (+ 383us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:08:24.171963 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:08:24.171971 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:08:24.172020 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:08:24.172464 (+ 444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:24.172864 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:24.173709 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:24.173917 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:24.173965 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:24.175620 (+ 1655us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 21:08:24.175622 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 21:08:24.175622 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 21:08:24.175627 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 21:08:24.176609 (+ 982us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:08:24.176897 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:08:24.176897 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:08:24.176897 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:08:24.176938 (+ 41us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:08:24.177103 (+ 165us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:08:24.177103 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:08:24.177103 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:08:24.177213 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 21:08:24.177256 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0924 21:08:24.177436 (+ 180us) server_negotiation.cc:299] Negotiation successful
0924 21:08:24.177487 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260924 21:08:24.177649 7010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:24.171313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44697 (local address 127.0.0.1:33596)
0924 21:08:24.171472 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0924 21:08:24.171505 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0924 21:08:24.171893 (+ 388us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 21:08:24.172062 (+ 169us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 21:08:24.172065 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 21:08:24.172261 (+ 196us) client_negotiation.cc:189] Negotiated authn=SASL
0924 21:08:24.172408 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:24.172411 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:24.172925 (+ 514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:24.172925 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:24.173645 (+ 720us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:24.173647 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:24.173976 (+ 329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:24.173977 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:24.174062 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:24.174398 (+ 336us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 21:08:24.174405 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0924 21:08:24.175540 (+ 1135us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 21:08:24.176668 (+ 1128us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 21:08:24.176673 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 21:08:24.176674 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 21:08:24.176841 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 21:08:24.176986 (+ 145us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 21:08:24.176987 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 21:08:24.176987 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 21:08:24.177062 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 21:08:24.177257 (+ 195us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 21:08:24.177260 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 21:08:24.177345 (+ 85us) client_negotiation.cc:769] Sending connection context
0924 21:08:24.177416 (+ 71us) client_negotiation.cc:240] Negotiation successful
0924 21:08:24.177505 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":94,"threads_started":1}
I20260924 21:08:24.178167 7008 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34067 } has no permanent_uuid. Determining permanent_uuid...
I20260924 21:08:24.178401 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAzNNjS/build/release/bin/kudu as pid 6940
Sep 24 21:08:24 dist-test-slave-c7tk krb5kdc[6431](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284104, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
Sep 24 21:08:24 dist-test-slave-c7tk krb5kdc[6431](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284103, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260924 21:08:24.185070 7010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:24.178402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:34067 (local address 127.0.0.1:60596)
0924 21:08:24.178593 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0924 21:08:24.178601 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0924 21:08:24.178634 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 21:08:24.178927 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 21:08:24.178928 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 21:08:24.179036 (+ 108us) client_negotiation.cc:189] Negotiated authn=SASL
0924 21:08:24.179074 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:24.179075 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:24.180053 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:24.180054 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:24.180582 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:24.180583 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:24.181222 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:24.181223 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:24.181278 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:24.181569 (+ 291us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 21:08:24.181573 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 21:08:24.182545 (+ 972us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 21:08:24.184138 (+ 1593us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 21:08:24.184138 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 21:08:24.184139 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 21:08:24.184266 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 21:08:24.184458 (+ 192us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 21:08:24.184459 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 21:08:24.184459 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 21:08:24.184499 (+ 40us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 21:08:24.184740 (+ 241us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 21:08:24.184740 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 21:08:24.184787 (+ 47us) client_negotiation.cc:769] Sending connection context
0924 21:08:24.184846 (+ 59us) client_negotiation.cc:240] Negotiation successful
0924 21:08:24.184929 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":180}
I20260924 21:08:24.185356 7011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:24.178459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60596 (local address 127.0.98.61:34067)
0924 21:08:24.178611 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0924 21:08:24.178615 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 21:08:24.178641 (+ 26us) server_negotiation.cc:407] Connection header received
0924 21:08:24.178695 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:08:24.178697 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:08:24.178708 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:08:24.178803 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:08:24.179153 (+ 350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:24.179961 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:24.180659 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:24.181154 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:24.181224 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:24.182613 (+ 1389us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 21:08:24.182619 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 21:08:24.182620 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 21:08:24.182626 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 21:08:24.184073 (+ 1447us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:08:24.184331 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:08:24.184333 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:08:24.184334 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:08:24.184393 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:08:24.184546 (+ 153us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:08:24.184547 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:08:24.184548 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:08:24.184682 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 21:08:24.184757 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0924 21:08:24.184912 (+ 155us) server_negotiation.cc:299] Negotiation successful
0924 21:08:24.184994 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":118,"thread_start_us":75,"threads_started":1}
I20260924 21:08:24.186472 7008 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a5110fb5b32e4973948078ef72376369: Bootstrap starting.
I20260924 21:08:24.186594 7013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:24.178816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60604 (local address 127.0.98.61:34067)
0924 21:08:24.178952 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0924 21:08:24.178955 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 21:08:24.178992 (+ 37us) server_negotiation.cc:407] Connection header received
0924 21:08:24.179064 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:08:24.179065 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:08:24.179074 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:08:24.179136 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:08:24.179668 (+ 532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:24.180251 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:24.181134 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:24.181438 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:24.181511 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:24.183688 (+ 2177us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 21:08:24.183689 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 21:08:24.183690 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 21:08:24.183695 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 21:08:24.185528 (+ 1833us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:08:24.185882 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:08:24.185883 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:08:24.185883 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:08:24.185932 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:08:24.186088 (+ 156us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:08:24.186089 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:08:24.186090 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:08:24.186183 (+ 93us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 21:08:24.186304 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0924 21:08:24.186360 (+ 56us) server_negotiation.cc:299] Negotiation successful
0924 21:08:24.186432 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118,"thread_start_us":78,"threads_started":1}
I20260924 21:08:24.187117 7008 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a5110fb5b32e4973948078ef72376369: Neither blocks nor log segments found. Creating new log.
I20260924 21:08:24.187388 7008 log.cc:824] T 00000000000000000000000000000000 P a5110fb5b32e4973948078ef72376369: Log is configured to *not* fsync() on all Append() calls
I20260924 21:08:24.187894 7008 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a5110fb5b32e4973948078ef72376369: No bootstrap required, opened a new log
I20260924 21:08:24.189137 7008 raft_consensus.cc:348] T 00000000000000000000000000000000 P a5110fb5b32e4973948078ef72376369 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f53f4b63d8e4da0905f8db79d1fc7db" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44697 } } peers { permanent_uuid: "a5110fb5b32e4973948078ef72376369" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34067 } }
I20260924 21:08:24.189239 7008 raft_consensus.cc:374] T 00000000000000000000000000000000 P a5110fb5b32e4973948078ef72376369 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 21:08:24.189285 7008 raft_consensus.cc:729] T 00000000000000000000000000000000 P a5110fb5b32e4973948078ef72376369 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5110fb5b32e4973948078ef72376369, State: Initialized, Role: FOLLOWER
I20260924 21:08:24.189399 7008 consensus_queue.cc:260] T 00000000000000000000000000000000 P a5110fb5b32e4973948078ef72376369 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f53f4b63d8e4da0905f8db79d1fc7db" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44697 } } peers { permanent_uuid: "a5110fb5b32e4973948078ef72376369" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34067 } }
I20260924 21:08:24.189626 7015 sys_catalog.cc:455] T 00000000000000000000000000000000 P a5110fb5b32e4973948078ef72376369 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f53f4b63d8e4da0905f8db79d1fc7db" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44697 } } peers { permanent_uuid: "a5110fb5b32e4973948078ef72376369" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34067 } } }
I20260924 21:08:24.189666 7008 sys_catalog.cc:564] T 00000000000000000000000000000000 P a5110fb5b32e4973948078ef72376369 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 21:08:24.189726 7015 sys_catalog.cc:458] T 00000000000000000000000000000000 P a5110fb5b32e4973948078ef72376369 [sys.catalog]: This master's current role is: FOLLOWER
W20260924 21:08:24.191280 7026 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a5110fb5b32e4973948078ef72376369: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 21:08:24.191349 7026 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 21:08:24.192572 6940 master_runner.cc:428] Detected that this master a5110fb5b32e4973948078ef72376369 is joining an existing cluster
I20260924 21:08:24.192607 6940 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.98.61:34067
I20260924 21:08:24.199942 7033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:24.193397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:34067 (local address 127.0.0.1:60614)
0924 21:08:24.193658 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0924 21:08:24.193666 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0924 21:08:24.193774 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 21:08:24.193915 (+ 141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 21:08:24.193915 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 21:08:24.194152 (+ 237us) client_negotiation.cc:189] Negotiated authn=SASL
0924 21:08:24.194216 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:24.194217 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:24.195041 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:24.195042 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:24.195816 (+ 774us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:24.195818 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:24.196263 (+ 445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:24.196263 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:24.196319 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:24.196667 (+ 348us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 21:08:24.196671 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 21:08:24.196975 (+ 304us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 21:08:24.198814 (+ 1839us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 21:08:24.198816 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 21:08:24.198816 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 21:08:24.199128 (+ 312us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 21:08:24.199296 (+ 168us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 21:08:24.199297 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 21:08:24.199297 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 21:08:24.199347 (+ 50us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 21:08:24.199602 (+ 255us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 21:08:24.199602 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 21:08:24.199648 (+ 46us) client_negotiation.cc:769] Sending connection context
0924 21:08:24.199751 (+ 103us) client_negotiation.cc:240] Negotiation successful
0924 21:08:24.199844 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":228,"mutex_wait_us":5,"thread_start_us":91,"threads_started":1}
I20260924 21:08:24.200019 6516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:24.193418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33610 (local address 127.0.98.62:44697)
0924 21:08:24.193489 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0924 21:08:24.193491 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 21:08:24.193753 (+ 262us) server_negotiation.cc:407] Connection header received
0924 21:08:24.193849 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:08:24.193849 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:08:24.193885 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:08:24.193913 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:08:24.194352 (+ 439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:24.195022 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:24.195648 (+ 626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:24.195822 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:24.195884 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:24.196762 (+ 878us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 21:08:24.196763 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 21:08:24.196764 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 21:08:24.196769 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 21:08:24.198766 (+ 1997us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:08:24.199142 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:08:24.199143 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:08:24.199143 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:08:24.199198 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:08:24.199439 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:08:24.199440 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:08:24.199440 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:08:24.199563 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 21:08:24.199642 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0924 21:08:24.199871 (+ 229us) server_negotiation.cc:299] Negotiation successful
0924 21:08:24.199930 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260924 21:08:24.200191 7032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:24.193539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44697 (local address 127.0.0.1:33610)
0924 21:08:24.193666 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0924 21:08:24.193689 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0924 21:08:24.193787 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 21:08:24.193967 (+ 180us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 21:08:24.193968 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 21:08:24.194188 (+ 220us) client_negotiation.cc:189] Negotiated authn=SASL
0924 21:08:24.194250 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:24.194252 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:24.195080 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:24.195081 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:24.195574 (+ 493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:24.195575 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:24.195897 (+ 322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:24.195898 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:24.195959 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:24.196329 (+ 370us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 21:08:24.196333 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 21:08:24.196686 (+ 353us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 21:08:24.198843 (+ 2157us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 21:08:24.198844 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 21:08:24.198844 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 21:08:24.199056 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 21:08:24.199271 (+ 215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 21:08:24.199272 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 21:08:24.199272 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 21:08:24.199345 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 21:08:24.199701 (+ 356us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 21:08:24.199702 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 21:08:24.199751 (+ 49us) client_negotiation.cc:769] Sending connection context
0924 21:08:24.199840 (+ 89us) client_negotiation.cc:240] Negotiation successful
0924 21:08:24.199919 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"mutex_wait_us":3,"thread_start_us":60,"threads_started":1}
I20260924 21:08:24.200438 7013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:24.193451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60614 (local address 127.0.98.61:34067)
0924 21:08:24.193496 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0924 21:08:24.193498 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 21:08:24.193748 (+ 250us) server_negotiation.cc:407] Connection header received
0924 21:08:24.193847 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:08:24.193848 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:08:24.193857 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:08:24.193902 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:08:24.194290 (+ 388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:24.194957 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:24.195902 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:24.196172 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:24.196206 (+ 34us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:24.197061 (+ 855us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 21:08:24.197063 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 21:08:24.197063 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 21:08:24.197069 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 21:08:24.198726 (+ 1657us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:08:24.199200 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:08:24.199201 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:08:24.199201 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:08:24.199244 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:08:24.199432 (+ 188us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:08:24.199432 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:08:24.199433 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:08:24.199529 (+ 96us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 21:08:24.199595 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0924 21:08:24.199770 (+ 175us) server_negotiation.cc:299] Negotiation successful
0924 21:08:24.200333 (+ 563us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":499,"server-negotiator.queue_time_us":29}
Sep 24 21:08:24 dist-test-slave-c7tk krb5kdc[6431](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284103, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260924 21:08:24.208319 7013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:24.202234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60630 (local address 127.0.98.61:34067)
0924 21:08:24.202268 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0924 21:08:24.202270 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 21:08:24.202380 (+ 110us) server_negotiation.cc:407] Connection header received
0924 21:08:24.202462 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:08:24.202463 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:08:24.202483 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:08:24.202531 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:08:24.202865 (+ 334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:24.203254 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:24.204003 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:24.204188 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:24.204260 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:24.206078 (+ 1818us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 21:08:24.206080 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 21:08:24.206080 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 21:08:24.206086 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 21:08:24.207294 (+ 1208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:08:24.207646 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:08:24.207647 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:08:24.207647 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:08:24.207697 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:08:24.207870 (+ 173us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:08:24.207871 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:08:24.207872 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:08:24.207968 (+ 96us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 21:08:24.208032 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0924 21:08:24.208183 (+ 151us) server_negotiation.cc:299] Negotiation successful
0924 21:08:24.208245 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260924 21:08:24.208351 7034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:24.202186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:34067 (local address 127.0.0.1:60630)
0924 21:08:24.202278 (+ 92us) negotiation.cc:107] Waiting for socket to connect
0924 21:08:24.202309 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0924 21:08:24.202394 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 21:08:24.202536 (+ 142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 21:08:24.202539 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 21:08:24.202757 (+ 218us) client_negotiation.cc:189] Negotiated authn=SASL
0924 21:08:24.202792 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:24.202793 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:24.203326 (+ 533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:24.203327 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:24.203915 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:24.203917 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:24.204263 (+ 346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:24.204264 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:24.204344 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:24.204767 (+ 423us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 21:08:24.204771 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 21:08:24.205987 (+ 1216us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 21:08:24.207390 (+ 1403us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 21:08:24.207395 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 21:08:24.207398 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 21:08:24.207571 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 21:08:24.207755 (+ 184us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 21:08:24.207755 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 21:08:24.207756 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 21:08:24.207806 (+ 50us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 21:08:24.208026 (+ 220us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 21:08:24.208031 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 21:08:24.208075 (+ 44us) client_negotiation.cc:769] Sending connection context
0924 21:08:24.208162 (+ 87us) client_negotiation.cc:240] Negotiation successful
0924 21:08:24.208253 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":68,"thread_start_us":47,"threads_started":1}
I20260924 21:08:24.208940 6476 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.98.61:34067: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "a5110fb5b32e4973948078ef72376369"
member_type: NON_VOTER
last_known_addr {
host: "127.0.98.61"
port: 34067
}
attrs {
promote: true
}
}
dest_uuid: "0f53f4b63d8e4da0905f8db79d1fc7db"
cas_config_opid_index: -1
I20260924 21:08:24.209209 6476 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0f53f4b63d8e4da0905f8db79d1fc7db [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "0f53f4b63d8e4da0905f8db79d1fc7db" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44697 } } peers { permanent_uuid: "a5110fb5b32e4973948078ef72376369" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 34067 } attrs { promote: true } }
I20260924 21:08:24.209610 7036 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 0f53f4b63d8e4da0905f8db79d1fc7db [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER a5110fb5b32e4973948078ef72376369 (127.0.98.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "0f53f4b63d8e4da0905f8db79d1fc7db" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44697 } } peers { permanent_uuid: "a5110fb5b32e4973948078ef72376369" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 34067 } attrs { promote: true } } }
I20260924 21:08:24.209900 6982 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a5110fb5b32e4973948078ef72376369 [term 0 FOLLOWER]: Advancing to term 1
I20260924 21:08:24.210453 7036 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.98.61:34067
I20260924 21:08:24.210624 6940 master.cc:537] Master@127.0.98.61:34067 shutting down...
I20260924 21:08:24.210645 6982 raft_consensus.cc:1261] T 00000000000000000000000000000000 P a5110fb5b32e4973948078ef72376369 [term 1 FOLLOWER]: Refusing update from remote peer 0f53f4b63d8e4da0905f8db79d1fc7db: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260924 21:08:24.210688 7035 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0f53f4b63d8e4da0905f8db79d1fc7db [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "0f53f4b63d8e4da0905f8db79d1fc7db" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "0f53f4b63d8e4da0905f8db79d1fc7db" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44697 } } peers { permanent_uuid: "a5110fb5b32e4973948078ef72376369" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 34067 } attrs { promote: true } } }
I20260924 21:08:24.210755 7035 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0f53f4b63d8e4da0905f8db79d1fc7db [sys.catalog]: This master's current role is: LEADER
I20260924 21:08:24.211676 7035 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 0f53f4b63d8e4da0905f8db79d1fc7db [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5110fb5b32e4973948078ef72376369" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 34067 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
W20260924 21:08:24.212247 6462 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
I20260924 21:08:24.213192 6940 raft_consensus.cc:2229] T 00000000000000000000000000000000 P a5110fb5b32e4973948078ef72376369 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 21:08:24.213263 6940 raft_consensus.cc:2258] T 00000000000000000000000000000000 P a5110fb5b32e4973948078ef72376369 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 21:08:24.213304 6940 tablet_replica.cc:330] T 00000000000000000000000000000000 P a5110fb5b32e4973948078ef72376369: stopping tablet replica
I20260924 21:08:24.218071 7034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:24.217987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:34067 (local address 127.0.0.1:60640)
0924 21:08:24.218028 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0924 21:08:24.218055 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:34067: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260924 21:08:24.218146 6462 consensus_peers.cc:487] T 00000000000000000000000000000000 P 0f53f4b63d8e4da0905f8db79d1fc7db -> Peer a5110fb5b32e4973948078ef72376369 (127.0.98.61:34067): Couldn't send request to peer a5110fb5b32e4973948078ef72376369. Status: Network error: Client connection negotiation failed: client connection to 127.0.98.61:34067: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260924 21:08:24.225977 6940 master.cc:552] Master@127.0.98.61:34067 shutdown complete.
I20260924 21:08:24.226016 6940 master_runner.cc:305] Clearing existing system tablet
I20260924 21:08:24.226118 6940 fs_manager.cc:359] Metadata directory not provided
I20260924 21:08:24.226157 6940 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 21:08:24.226722 6940 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.001s sys 0.000s
I20260924 21:08:24.227144 7040 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 21:08:24.227262 6940 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790284076800039-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 21:08:24.227313 6940 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 21:08:24.227334 6940 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790284076800039-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790284076800039-392-0/minicluster-data/master-1/wal
uuid: "a5110fb5b32e4973948078ef72376369"
format_stamp: "Formatted at 2026-09-24 21:08:24 on dist-test-slave-c7tk"
server_key: "f28599b3a36446a50e9c15952a945a86"
server_key_iv: "b41fee4f49a573c746eb3406870b7e7f"
server_key_version: "encryptionkey@0"
I20260924 21:08:24.227921 6940 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P a5110fb5b32e4973948078ef72376369: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 21:08:24.229231 6940 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P a5110fb5b32e4973948078ef72376369: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260924 21:08:24.229296 6940 log.cc:1196] T 00000000000000000000000000000000 P a5110fb5b32e4973948078ef72376369: Deleting WAL directory at /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790284076800039-392-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260924 21:08:24.229480 6940 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P a5110fb5b32e4973948078ef72376369: Deleting consensus metadata
I20260924 21:08:24.229592 6940 master_runner.cc:315] Copying system tablet from 127.0.98.62:44697
I20260924 21:08:24.230149 6940 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P a5110fb5b32e4973948078ef72376369: tablet copy: Beginning tablet copy session from remote peer at address 127.0.98.62:44697
I20260924 21:08:24.235221 6516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:24.230423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33624 (local address 127.0.98.62:44697)
0924 21:08:24.230454 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0924 21:08:24.230456 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 21:08:24.230501 (+ 45us) server_negotiation.cc:407] Connection header received
0924 21:08:24.230555 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:08:24.230556 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:08:24.230562 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:08:24.230597 (+ 35us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:08:24.230846 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:24.231252 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:24.232129 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:24.232278 (+ 149us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:24.232345 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:24.233106 (+ 761us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 21:08:24.233107 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 21:08:24.233107 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 21:08:24.233112 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 21:08:24.234365 (+ 1253us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:08:24.234695 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:08:24.234696 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:08:24.234696 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:08:24.234736 (+ 40us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:08:24.234901 (+ 165us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:08:24.234901 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:08:24.234901 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:08:24.234972 (+ 71us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 21:08:24.235013 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0924 21:08:24.235139 (+ 126us) server_negotiation.cc:299] Negotiation successful
0924 21:08:24.235179 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260924 21:08:24.235298 7046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:24.230350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44697 (local address 127.0.0.1:33624)
0924 21:08:24.230450 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0924 21:08:24.230458 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0924 21:08:24.230508 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 21:08:24.230608 (+ 100us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 21:08:24.230609 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 21:08:24.230744 (+ 135us) client_negotiation.cc:189] Negotiated authn=SASL
0924 21:08:24.230787 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:24.230788 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:24.231316 (+ 528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:24.231317 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:24.232061 (+ 744us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:24.232063 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:24.232346 (+ 283us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:24.232347 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:24.232417 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:24.232711 (+ 294us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 21:08:24.232715 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 21:08:24.233026 (+ 311us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 21:08:24.234437 (+ 1411us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 21:08:24.234438 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 21:08:24.234438 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 21:08:24.234627 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 21:08:24.234787 (+ 160us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 21:08:24.234788 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 21:08:24.234789 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 21:08:24.234851 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 21:08:24.235024 (+ 173us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 21:08:24.235025 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 21:08:24.235062 (+ 37us) client_negotiation.cc:769] Sending connection context
0924 21:08:24.235128 (+ 66us) client_negotiation.cc:240] Negotiation successful
0924 21:08:24.235205 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":84,"thread_start_us":69,"threads_started":1}
I20260924 21:08:24.235550 6496 tablet_copy_service.cc:140] P 0f53f4b63d8e4da0905f8db79d1fc7db: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer a5110fb5b32e4973948078ef72376369 ({username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:33624)
I20260924 21:08:24.235623 6496 tablet_copy_service.cc:161] P 0f53f4b63d8e4da0905f8db79d1fc7db: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer a5110fb5b32e4973948078ef72376369 at {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:33624: session id = a5110fb5b32e4973948078ef72376369-00000000000000000000000000000000
I20260924 21:08:24.236276 6496 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 0f53f4b63d8e4da0905f8db79d1fc7db: Tablet Copy: opened 0 blocks and 1 log segments
I20260924 21:08:24.236769 6940 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260924 21:08:24.237658 6940 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P a5110fb5b32e4973948078ef72376369: tablet copy: Starting download of 0 data blocks...
I20260924 21:08:24.237773 6940 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P a5110fb5b32e4973948078ef72376369: tablet copy: Starting download of 1 WAL segments...
I20260924 21:08:24.238466 6940 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P a5110fb5b32e4973948078ef72376369: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260924 21:08:24.239272 6496 tablet_copy_service.cc:342] P 0f53f4b63d8e4da0905f8db79d1fc7db: Request end of tablet copy session a5110fb5b32e4973948078ef72376369-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:33624
I20260924 21:08:24.239326 6496 tablet_copy_service.cc:433] P 0f53f4b63d8e4da0905f8db79d1fc7db: ending tablet copy session a5110fb5b32e4973948078ef72376369-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer a5110fb5b32e4973948078ef72376369
I20260924 21:08:24.241170 6940 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 21:08:24.242244 7048 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 21:08:24.242240 7047 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 21:08:24.246595 6940 server_base.cc:1034] running on GCE node
W20260924 21:08:24.246615 7050 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 21:08:24.246737 6940 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 21:08:24.246857 6940 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 21:08:24.247984 6940 hybrid_clock.cc:648] HybridClock initialized: now 1790284104247974 us; error 22 us; skew 500 ppm
Sep 24 21:08:24 dist-test-slave-c7tk krb5kdc[6431](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284104, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 21:08:24.249107 6940 init.cc:377] Logged in from keytab as oryx/127.0.98.61@KRBTEST.COM (short username oryx)
I20260924 21:08:24.249369 6940 webserver.cc:460] Webserver started at http://127.0.98.61:42357/ using document root <none> and password file <none>
I20260924 21:08:24.249486 6940 fs_manager.cc:359] Metadata directory not provided
I20260924 21:08:24.249543 6940 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 21:08:24.250141 6940 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.000s sys 0.001s
I20260924 21:08:24.250653 7057 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 21:08:24.250784 6940 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 21:08:24.250845 6940 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790284076800039-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790284076800039-392-0/minicluster-data/master-1/wal
uuid: "a5110fb5b32e4973948078ef72376369"
format_stamp: "Formatted at 2026-09-24 21:08:24 on dist-test-slave-c7tk"
server_key: "f28599b3a36446a50e9c15952a945a86"
server_key_iv: "b41fee4f49a573c746eb3406870b7e7f"
server_key_version: "encryptionkey@0"
I20260924 21:08:24.250893 6940 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790284076800039-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790284076800039-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790284076800039-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 21:08:24.259950 6940 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 21:08:24.261438 6940 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 21:08:24.264174 6940 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:34067
I20260924 21:08:24.264372 7109 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:34067 every 8 connection(s)
I20260924 21:08:24.264461 6940 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790284076800039-392-0/minicluster-data/master-1/data/info.pb
I20260924 21:08:24.264995 7110 sys_catalog.cc:263] Verifying existing consensus state
I20260924 21:08:24.265327 7110 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a5110fb5b32e4973948078ef72376369: Bootstrap starting.
I20260924 21:08:24.267225 7110 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a5110fb5b32e4973948078ef72376369: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 21:08:24.267422 7110 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a5110fb5b32e4973948078ef72376369: Bootstrap complete.
I20260924 21:08:24.267581 7110 raft_consensus.cc:348] T 00000000000000000000000000000000 P a5110fb5b32e4973948078ef72376369 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "0f53f4b63d8e4da0905f8db79d1fc7db" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44697 } } peers { permanent_uuid: "a5110fb5b32e4973948078ef72376369" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 34067 } attrs { promote: true } }
I20260924 21:08:24.267630 7110 raft_consensus.cc:729] T 00000000000000000000000000000000 P a5110fb5b32e4973948078ef72376369 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: a5110fb5b32e4973948078ef72376369, State: Initialized, Role: LEARNER
I20260924 21:08:24.267655 7110 consensus_queue.cc:260] T 00000000000000000000000000000000 P a5110fb5b32e4973948078ef72376369 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "0f53f4b63d8e4da0905f8db79d1fc7db" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44697 } } peers { permanent_uuid: "a5110fb5b32e4973948078ef72376369" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 34067 } attrs { promote: true } }
I20260924 21:08:24.267817 7110 sys_catalog.cc:564] T 00000000000000000000000000000000 P a5110fb5b32e4973948078ef72376369 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 21:08:24.267825 7113 sys_catalog.cc:455] T 00000000000000000000000000000000 P a5110fb5b32e4973948078ef72376369 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "0f53f4b63d8e4da0905f8db79d1fc7db" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44697 } } peers { permanent_uuid: "a5110fb5b32e4973948078ef72376369" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 34067 } attrs { promote: true } } }
I20260924 21:08:24.267930 7113 sys_catalog.cc:458] T 00000000000000000000000000000000 P a5110fb5b32e4973948078ef72376369 [sys.catalog]: This master's current role is: LEARNER
I20260924 21:08:24.269119 7124 catalog_manager.cc:1259] Loaded cluster ID: e2489afc08c64420b12bd07cdab5f107
I20260924 21:08:24.269168 7124 catalog_manager.cc:1552] T 00000000000000000000000000000000 P a5110fb5b32e4973948078ef72376369: loading cluster ID for follower catalog manager: success
I20260924 21:08:24.270227 7124 catalog_manager.cc:1574] T 00000000000000000000000000000000 P a5110fb5b32e4973948078ef72376369: acquiring CA information for follower catalog manager: success
I20260924 21:08:24.270527 7124 catalog_manager.cc:1602] T 00000000000000000000000000000000 P a5110fb5b32e4973948078ef72376369: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 21:08:24.294219 7125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:24.288052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60658 (local address 127.0.98.61:34067)
0924 21:08:24.288153 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0924 21:08:24.288155 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 21:08:24.288162 (+ 7us) server_negotiation.cc:407] Connection header received
0924 21:08:24.288183 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:08:24.288183 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:08:24.288192 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:08:24.288750 (+ 558us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:08:24.288776 (+ 26us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:24.289402 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:24.290321 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:24.290577 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:24.290674 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:24.291743 (+ 1069us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 21:08:24.291745 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 21:08:24.291745 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 21:08:24.291751 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 21:08:24.293069 (+ 1318us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:08:24.293391 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:08:24.293392 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:08:24.293393 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:08:24.293458 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:08:24.293681 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:08:24.293682 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:08:24.293682 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:08:24.293780 (+ 98us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 21:08:24.293941 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0924 21:08:24.294040 (+ 99us) server_negotiation.cc:299] Negotiation successful
0924 21:08:24.294106 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83,"thread_start_us":41,"threads_started":1}
WARNING: no policy specified for oryx/127.0.98.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.60@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790284076800039-392-0/krb5kdc/oryx_127.0.98.60.keytab.
Entry for principal oryx/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790284076800039-392-0/krb5kdc/oryx_127.0.98.60.keytab.
I20260924 21:08:24.316589 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAzNNjS/build/release/bin/kudu
/tmp/dist-test-taskAzNNjS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790284076800039-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790284076800039-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790284076800039-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790284076800039-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:39715
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790284076800039-392-0/krb5kdc/oryx_127.0.98.60.keytab
--principal=oryx/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37913
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44697,127.0.98.61:34067,127.0.98.60:39715
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790284076800039-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790284076800039-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790284076800039-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 21:08:24.394481 7128 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 21:08:24.394644 7128 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 21:08:24.394662 7128 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 21:08:24.395623 7128 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 21:08:24.395658 7128 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 21:08:24.395697 7128 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 21:08:24.395723 7128 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 21:08:24.395740 7128 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 21:08:24.396996 7128 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37913
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790284076800039-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790284076800039-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44697,127.0.98.61:34067,127.0.98.60:39715
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790284076800039-392-0/krb5kdc/oryx_127.0.98.60.keytab
--principal=oryx/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:39715
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790284076800039-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7128
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790284076800039-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 19:49:06 UTC on ccycloud.quasar-vhqjoc.root.comops.site
build id 4670864
I20260924 21:08:24.397212 7128 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 21:08:24.397408 7128 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 21:08:24.401607 7134 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 21:08:24.401727 7136 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 21:08:24.401640 7133 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 21:08:24.401763 7128 server_base.cc:1034] running on GCE node
I20260924 21:08:24.402097 7128 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 21:08:24.402367 7128 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 21:08:24.403514 7128 hybrid_clock.cc:648] HybridClock initialized: now 1790284104403503 us; error 27 us; skew 500 ppm
Sep 24 21:08:24 dist-test-slave-c7tk krb5kdc[6431](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284104, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 21:08:24.405686 7128 init.cc:377] Logged in from keytab as oryx/127.0.98.60@KRBTEST.COM (short username oryx)
I20260924 21:08:24.406085 7128 webserver.cc:460] Webserver started at http://127.0.98.60:35491/ using document root <none> and password file <none>
I20260924 21:08:24.406267 7128 fs_manager.cc:359] Metadata directory not provided
I20260924 21:08:24.406324 7128 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 21:08:24.406461 7128 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 21:08:24.407279 7128 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790284076800039-392-0/minicluster-data/master-2/data/instance:
uuid: "65393c0abc864ec4b9d3d76877a9a863"
format_stamp: "Formatted at 2026-09-24 21:08:24 on dist-test-slave-c7tk"
server_key: "929801a3804c76335e16b5f9c98a0a5a"
server_key_iv: "8b4f4eb4f212052792984ddcb255de39"
server_key_version: "encryptionkey@0"
I20260924 21:08:24.407591 7128 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790284076800039-392-0/minicluster-data/master-2/wal/instance:
uuid: "65393c0abc864ec4b9d3d76877a9a863"
format_stamp: "Formatted at 2026-09-24 21:08:24 on dist-test-slave-c7tk"
server_key: "929801a3804c76335e16b5f9c98a0a5a"
server_key_iv: "8b4f4eb4f212052792984ddcb255de39"
server_key_version: "encryptionkey@0"
I20260924 21:08:24.408747 7128 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 21:08:24.409466 7143 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 21:08:24.409642 7128 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 21:08:24.409734 7128 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790284076800039-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790284076800039-392-0/minicluster-data/master-2/wal
uuid: "65393c0abc864ec4b9d3d76877a9a863"
format_stamp: "Formatted at 2026-09-24 21:08:24 on dist-test-slave-c7tk"
server_key: "929801a3804c76335e16b5f9c98a0a5a"
server_key_iv: "8b4f4eb4f212052792984ddcb255de39"
server_key_version: "encryptionkey@0"
I20260924 21:08:24.409853 7128 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790284076800039-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790284076800039-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790284076800039-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 21:08:24.427085 7128 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 21:08:24.428524 7128 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 21:08:24.428699 7128 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 21:08:24.433075 7128 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:39715
I20260924 21:08:24.433101 7195 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:39715 every 8 connection(s)
I20260924 21:08:24.433411 7128 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790284076800039-392-0/minicluster-data/master-2/data/info.pb
I20260924 21:08:24.434306 7196 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 21:08:24.435525 7196 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44697 } has no permanent_uuid. Determining permanent_uuid...
I20260924 21:08:24.440500 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAzNNjS/build/release/bin/kudu as pid 7128
Sep 24 21:08:24 dist-test-slave-c7tk krb5kdc[6431](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284104, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260924 21:08:24.443697 6516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:24.436142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33638 (local address 127.0.98.62:44697)
0924 21:08:24.436212 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0924 21:08:24.436214 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 21:08:24.436311 (+ 97us) server_negotiation.cc:407] Connection header received
0924 21:08:24.436735 (+ 424us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:08:24.436736 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:08:24.436747 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:08:24.436804 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:08:24.437325 (+ 521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:24.437769 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:24.438726 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:24.438920 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:24.439006 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:24.441076 (+ 2070us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 21:08:24.441077 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 21:08:24.441077 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 21:08:24.441085 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 21:08:24.442557 (+ 1472us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:08:24.442886 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:08:24.442887 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:08:24.442887 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:08:24.442945 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:08:24.443178 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:08:24.443179 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:08:24.443179 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:08:24.443265 (+ 86us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 21:08:24.443308 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0924 21:08:24.443572 (+ 264us) server_negotiation.cc:299] Negotiation successful
0924 21:08:24.443626 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260924 21:08:24.443912 7198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:24.436029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44697 (local address 127.0.0.1:33638)
0924 21:08:24.436198 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0924 21:08:24.436238 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 21:08:24.436645 (+ 407us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 21:08:24.436875 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 21:08:24.436880 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 21:08:24.437086 (+ 206us) client_negotiation.cc:189] Negotiated authn=SASL
0924 21:08:24.437236 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:24.437240 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:24.437907 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:24.437907 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:24.438627 (+ 720us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:24.438629 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:24.439026 (+ 397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:24.439027 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:24.439137 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:24.439524 (+ 387us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 21:08:24.439531 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0924 21:08:24.440979 (+ 1448us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 21:08:24.442630 (+ 1651us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 21:08:24.442635 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 21:08:24.442636 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 21:08:24.442820 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 21:08:24.443003 (+ 183us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 21:08:24.443004 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 21:08:24.443004 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 21:08:24.443120 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 21:08:24.443320 (+ 200us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 21:08:24.443324 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 21:08:24.443459 (+ 135us) client_negotiation.cc:769] Sending connection context
0924 21:08:24.443554 (+ 95us) client_negotiation.cc:240] Negotiation successful
0924 21:08:24.443699 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":73,"threads_started":1}
I20260924 21:08:24.444406 7196 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34067 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 21:08:24 dist-test-slave-c7tk krb5kdc[6431](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284103, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
Sep 24 21:08:24 dist-test-slave-c7tk krb5kdc[6431](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284104, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260924 21:08:24.449710 7199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:24.441021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57282 (local address 127.0.98.60:39715)
0924 21:08:24.441145 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0924 21:08:24.441149 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 21:08:24.441167 (+ 18us) server_negotiation.cc:407] Connection header received
0924 21:08:24.441186 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:08:24.441187 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:08:24.441199 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:08:24.441271 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:08:24.441952 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:24.442841 (+ 889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:24.443826 (+ 985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:24.444249 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:24.444309 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:24.446782 (+ 2473us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 21:08:24.446789 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 21:08:24.446789 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 21:08:24.446795 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 21:08:24.448413 (+ 1618us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:08:24.448815 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:08:24.448818 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:08:24.448818 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:08:24.448896 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:08:24.449139 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:08:24.449140 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:08:24.449140 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:08:24.449284 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 21:08:24.449399 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0924 21:08:24.449506 (+ 107us) server_negotiation.cc:299] Negotiation successful
0924 21:08:24.449560 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97,"thread_start_us":74,"threads_started":1}
I20260924 21:08:24.450899 7198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:24.444650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:34067 (local address 127.0.0.1:60660)
0924 21:08:24.444777 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0924 21:08:24.444787 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 21:08:24.444828 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 21:08:24.445008 (+ 180us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 21:08:24.445009 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 21:08:24.445148 (+ 139us) client_negotiation.cc:189] Negotiated authn=SASL
0924 21:08:24.445199 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:24.445201 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:24.445828 (+ 627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:24.445829 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:24.446367 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:24.446369 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:24.446774 (+ 405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:24.446775 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:24.446845 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:24.447158 (+ 313us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 21:08:24.447161 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0924 21:08:24.448172 (+ 1011us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 21:08:24.449902 (+ 1730us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 21:08:24.449903 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 21:08:24.449904 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 21:08:24.450113 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 21:08:24.450307 (+ 194us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 21:08:24.450308 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 21:08:24.450308 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 21:08:24.450372 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 21:08:24.450618 (+ 246us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 21:08:24.450618 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 21:08:24.450656 (+ 38us) client_negotiation.cc:769] Sending connection context
0924 21:08:24.450717 (+ 61us) client_negotiation.cc:240] Negotiation successful
0924 21:08:24.450797 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114}
I20260924 21:08:24.451180 7125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:24.444734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60660 (local address 127.0.98.61:34067)
0924 21:08:24.444799 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0924 21:08:24.444801 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 21:08:24.444818 (+ 17us) server_negotiation.cc:407] Connection header received
0924 21:08:24.444927 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:08:24.444928 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:08:24.444937 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:08:24.444981 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:08:24.445273 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:24.445721 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:24.446477 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:24.446682 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:24.446775 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:24.448253 (+ 1478us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 21:08:24.448254 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 21:08:24.448254 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 21:08:24.448264 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0924 21:08:24.449806 (+ 1542us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:08:24.450174 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:08:24.450175 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:08:24.450176 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:08:24.450248 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:08:24.450443 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:08:24.450444 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:08:24.450445 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:08:24.450555 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 21:08:24.450626 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0924 21:08:24.451013 (+ 387us) server_negotiation.cc:299] Negotiation successful
0924 21:08:24.451094 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260924 21:08:24.451462 7196 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39715 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 21:08:24 dist-test-slave-c7tk krb5kdc[6431](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284104, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260924 21:08:24.456516 7198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:24.451651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39715 (local address 127.0.0.1:57286)
0924 21:08:24.451695 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0924 21:08:24.451705 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 21:08:24.451743 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 21:08:24.451891 (+ 148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 21:08:24.451891 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 21:08:24.451985 (+ 94us) client_negotiation.cc:189] Negotiated authn=SASL
0924 21:08:24.452008 (+ 23us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:24.452009 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:24.452539 (+ 530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:24.452539 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:24.452962 (+ 423us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:24.452963 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:24.453240 (+ 277us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:24.453241 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:24.453272 (+ 31us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:24.453542 (+ 270us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 21:08:24.453545 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0924 21:08:24.454571 (+ 1026us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 21:08:24.455946 (+ 1375us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 21:08:24.455947 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 21:08:24.455947 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 21:08:24.456069 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 21:08:24.456190 (+ 121us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 21:08:24.456190 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 21:08:24.456190 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 21:08:24.456225 (+ 35us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 21:08:24.456350 (+ 125us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 21:08:24.456351 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 21:08:24.456377 (+ 26us) client_negotiation.cc:769] Sending connection context
0924 21:08:24.456416 (+ 39us) client_negotiation.cc:240] Negotiation successful
0924 21:08:24.456471 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260924 21:08:24.456589 7199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:24.451717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57286 (local address 127.0.98.60:39715)
0924 21:08:24.451745 (+ 28us) server_negotiation.cc:206] Beginning negotiation
0924 21:08:24.451746 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 21:08:24.451784 (+ 38us) server_negotiation.cc:407] Connection header received
0924 21:08:24.451826 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:08:24.451827 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:08:24.451835 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:08:24.451889 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:08:24.452063 (+ 174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:24.452472 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:24.453021 (+ 549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:24.453185 (+ 164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:24.453230 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:24.454642 (+ 1412us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 21:08:24.454643 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 21:08:24.454643 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 21:08:24.454648 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 21:08:24.455889 (+ 1241us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:08:24.456116 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:08:24.456116 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:08:24.456117 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:08:24.456151 (+ 34us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:08:24.456255 (+ 104us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:08:24.456255 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:08:24.456255 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:08:24.456320 (+ 65us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 21:08:24.456357 (+ 37us) server_negotiation.cc:1035] Waiting for connection context
0924 21:08:24.456430 (+ 73us) server_negotiation.cc:299] Negotiation successful
0924 21:08:24.456507 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":31,"server-negotiator.queue_time_us":14}
I20260924 21:08:24.457649 7196 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 65393c0abc864ec4b9d3d76877a9a863: Bootstrap starting.
I20260924 21:08:24.458441 7196 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 65393c0abc864ec4b9d3d76877a9a863: Neither blocks nor log segments found. Creating new log.
I20260924 21:08:24.458738 7196 log.cc:824] T 00000000000000000000000000000000 P 65393c0abc864ec4b9d3d76877a9a863: Log is configured to *not* fsync() on all Append() calls
I20260924 21:08:24.459350 7196 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 65393c0abc864ec4b9d3d76877a9a863: No bootstrap required, opened a new log
I20260924 21:08:24.460595 7196 raft_consensus.cc:348] T 00000000000000000000000000000000 P 65393c0abc864ec4b9d3d76877a9a863 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f53f4b63d8e4da0905f8db79d1fc7db" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44697 } } peers { permanent_uuid: "a5110fb5b32e4973948078ef72376369" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34067 } } peers { permanent_uuid: "65393c0abc864ec4b9d3d76877a9a863" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39715 } }
I20260924 21:08:24.460718 7196 raft_consensus.cc:374] T 00000000000000000000000000000000 P 65393c0abc864ec4b9d3d76877a9a863 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 21:08:24.460758 7196 raft_consensus.cc:729] T 00000000000000000000000000000000 P 65393c0abc864ec4b9d3d76877a9a863 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65393c0abc864ec4b9d3d76877a9a863, State: Initialized, Role: FOLLOWER
I20260924 21:08:24.460870 7196 consensus_queue.cc:260] T 00000000000000000000000000000000 P 65393c0abc864ec4b9d3d76877a9a863 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f53f4b63d8e4da0905f8db79d1fc7db" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44697 } } peers { permanent_uuid: "a5110fb5b32e4973948078ef72376369" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34067 } } peers { permanent_uuid: "65393c0abc864ec4b9d3d76877a9a863" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39715 } }
I20260924 21:08:24.461163 7196 sys_catalog.cc:564] T 00000000000000000000000000000000 P 65393c0abc864ec4b9d3d76877a9a863 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 21:08:24.461153 7201 sys_catalog.cc:455] T 00000000000000000000000000000000 P 65393c0abc864ec4b9d3d76877a9a863 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f53f4b63d8e4da0905f8db79d1fc7db" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44697 } } peers { permanent_uuid: "a5110fb5b32e4973948078ef72376369" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34067 } } peers { permanent_uuid: "65393c0abc864ec4b9d3d76877a9a863" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39715 } } }
I20260924 21:08:24.461259 7201 sys_catalog.cc:458] T 00000000000000000000000000000000 P 65393c0abc864ec4b9d3d76877a9a863 [sys.catalog]: This master's current role is: FOLLOWER
W20260924 21:08:24.462725 7212 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 65393c0abc864ec4b9d3d76877a9a863: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 21:08:24.462777 7212 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 21:08:24.465517 7128 master_runner.cc:428] Detected that this master 65393c0abc864ec4b9d3d76877a9a863 is joining an existing cluster
I20260924 21:08:24.465555 7128 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.98.60:39715
I20260924 21:08:24.472847 7218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:24.466491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39715 (local address 127.0.0.1:57288)
0924 21:08:24.466733 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0924 21:08:24.466742 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 21:08:24.466821 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 21:08:24.466966 (+ 145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 21:08:24.466967 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 21:08:24.467185 (+ 218us) client_negotiation.cc:189] Negotiated authn=SASL
0924 21:08:24.467256 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:24.467257 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:24.468112 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:24.468112 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:24.468911 (+ 799us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:24.468913 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:24.469352 (+ 439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:24.469353 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:24.469412 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:24.470199 (+ 787us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 21:08:24.470202 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0924 21:08:24.470451 (+ 249us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 21:08:24.471822 (+ 1371us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 21:08:24.471822 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 21:08:24.471823 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 21:08:24.471980 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 21:08:24.472170 (+ 190us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 21:08:24.472171 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 21:08:24.472171 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 21:08:24.472231 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 21:08:24.472486 (+ 255us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 21:08:24.472487 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 21:08:24.472554 (+ 67us) client_negotiation.cc:769] Sending connection context
0924 21:08:24.472638 (+ 84us) client_negotiation.cc:240] Negotiation successful
0924 21:08:24.472730 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":85,"threads_started":1}
I20260924 21:08:24.472905 7199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:24.466655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57288 (local address 127.0.98.60:39715)
0924 21:08:24.466686 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0924 21:08:24.466688 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 21:08:24.466807 (+ 119us) server_negotiation.cc:407] Connection header received
0924 21:08:24.466869 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:08:24.466870 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:08:24.466879 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:08:24.466930 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:08:24.467326 (+ 396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:24.468037 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:24.468987 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:24.469254 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:24.469285 (+ 31us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:24.470560 (+ 1275us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 21:08:24.470561 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 21:08:24.470562 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 21:08:24.470568 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 21:08:24.471733 (+ 1165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:08:24.472056 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:08:24.472057 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:08:24.472057 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:08:24.472101 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:08:24.472297 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:08:24.472298 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:08:24.472298 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:08:24.472410 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 21:08:24.472493 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0924 21:08:24.472659 (+ 166us) server_negotiation.cc:299] Negotiation successful
0924 21:08:24.472778 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":62,"server-negotiator.queue_time_us":16}
I20260924 21:08:24.489135 7219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:24.466521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:34067 (local address 127.0.0.1:60666)
0924 21:08:24.466902 (+ 381us) negotiation.cc:107] Waiting for socket to connect
0924 21:08:24.466911 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 21:08:24.466956 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 21:08:24.467117 (+ 161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 21:08:24.467118 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 21:08:24.467384 (+ 266us) client_negotiation.cc:189] Negotiated authn=SASL
0924 21:08:24.467438 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:24.467440 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:24.468417 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:24.468417 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:24.469255 (+ 838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:24.469257 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:24.469854 (+ 597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:24.469854 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:24.469909 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:24.486237 (+ 16328us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 21:08:24.486241 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 21:08:24.486585 (+ 344us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 21:08:24.488124 (+ 1539us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 21:08:24.488124 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 21:08:24.488125 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 21:08:24.488269 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 21:08:24.488470 (+ 201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 21:08:24.488471 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 21:08:24.488471 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 21:08:24.488538 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 21:08:24.488834 (+ 296us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 21:08:24.488834 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 21:08:24.488876 (+ 42us) client_negotiation.cc:769] Sending connection context
0924 21:08:24.488943 (+ 67us) client_negotiation.cc:240] Negotiation successful
0924 21:08:24.489031 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":369,"thread_start_us":69,"threads_started":1}
I20260924 21:08:24.489138 7125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:24.466679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60666 (local address 127.0.98.61:34067)
0924 21:08:24.466715 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0924 21:08:24.466717 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 21:08:24.466968 (+ 251us) server_negotiation.cc:407] Connection header received
0924 21:08:24.466996 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:08:24.466997 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:08:24.467006 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:08:24.467074 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:08:24.467525 (+ 451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:24.468254 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:24.469351 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:24.469674 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:24.469717 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:24.486670 (+ 16953us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 21:08:24.486671 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 21:08:24.486672 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 21:08:24.486678 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 21:08:24.488040 (+ 1362us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:08:24.488362 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:08:24.488363 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:08:24.488364 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:08:24.488405 (+ 41us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:08:24.488641 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:08:24.488642 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:08:24.488642 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:08:24.488769 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 21:08:24.488832 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0924 21:08:24.488966 (+ 134us) server_negotiation.cc:299] Negotiation successful
0924 21:08:24.489046 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 21:08:24.489387 6516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:24.466491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33646 (local address 127.0.98.62:44697)
0924 21:08:24.466674 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0924 21:08:24.466676 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 21:08:24.467098 (+ 422us) server_negotiation.cc:407] Connection header received
0924 21:08:24.467184 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:08:24.467185 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:08:24.467195 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:08:24.467226 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:08:24.467645 (+ 419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:24.468370 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:24.469425 (+ 1055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:24.469707 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:24.469745 (+ 38us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:24.486760 (+ 17015us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 21:08:24.486761 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 21:08:24.486762 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 21:08:24.486768 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 21:08:24.488275 (+ 1507us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:08:24.488628 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:08:24.488629 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:08:24.488629 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:08:24.488689 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:08:24.488885 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:08:24.488886 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:08:24.488887 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:08:24.489011 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 21:08:24.489074 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0924 21:08:24.489254 (+ 180us) server_negotiation.cc:299] Negotiation successful
0924 21:08:24.489326 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":159}
I20260924 21:08:24.489702 7220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:24.466543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44697 (local address 127.0.0.1:33646)
0924 21:08:24.467039 (+ 496us) negotiation.cc:107] Waiting for socket to connect
0924 21:08:24.467048 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 21:08:24.467119 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 21:08:24.467281 (+ 162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 21:08:24.467282 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 21:08:24.467480 (+ 198us) client_negotiation.cc:189] Negotiated authn=SASL
0924 21:08:24.467532 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:24.467534 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:24.468521 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:24.468522 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:24.469326 (+ 804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:24.469328 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:24.469890 (+ 562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:24.469891 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:24.469947 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:24.486368 (+ 16421us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 21:08:24.486372 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 21:08:24.486685 (+ 313us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 21:08:24.488350 (+ 1665us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 21:08:24.488350 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 21:08:24.488350 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 21:08:24.488538 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 21:08:24.488759 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 21:08:24.488760 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 21:08:24.488760 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 21:08:24.488826 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 21:08:24.489099 (+ 273us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 21:08:24.489100 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 21:08:24.489138 (+ 38us) client_negotiation.cc:769] Sending connection context
0924 21:08:24.489509 (+ 371us) client_negotiation.cc:240] Negotiation successful
0924 21:08:24.489609 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":476,"thread_start_us":58,"threads_started":1}
Sep 24 21:08:24 dist-test-slave-c7tk krb5kdc[6431](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284103, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260924 21:08:24.496685 7199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:24.491230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57292 (local address 127.0.98.60:39715)
0924 21:08:24.491260 (+ 30us) server_negotiation.cc:206] Beginning negotiation
0924 21:08:24.491262 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 21:08:24.491269 (+ 7us) server_negotiation.cc:407] Connection header received
0924 21:08:24.491322 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:08:24.491322 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:08:24.491330 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:08:24.491382 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:08:24.491602 (+ 220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:24.492034 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:24.492692 (+ 658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:24.492896 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:24.492960 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:24.494556 (+ 1596us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 21:08:24.494557 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 21:08:24.494558 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 21:08:24.494564 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 21:08:24.495709 (+ 1145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:08:24.496013 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:08:24.496015 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:08:24.496015 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:08:24.496068 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:08:24.496247 (+ 179us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:08:24.496248 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:08:24.496248 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:08:24.496345 (+ 97us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 21:08:24.496420 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0924 21:08:24.496543 (+ 123us) server_negotiation.cc:299] Negotiation successful
0924 21:08:24.496607 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260924 21:08:24.496691 7034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:24.491182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39715 (local address 127.0.0.1:57292)
0924 21:08:24.491224 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0924 21:08:24.491232 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0924 21:08:24.491264 (+ 32us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 21:08:24.491386 (+ 122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 21:08:24.491386 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 21:08:24.491507 (+ 121us) client_negotiation.cc:189] Negotiated authn=SASL
0924 21:08:24.491534 (+ 27us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:24.491536 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:24.492117 (+ 581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:24.492118 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:24.492607 (+ 489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:24.492608 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:24.492959 (+ 351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:24.492960 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:24.493019 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:24.493383 (+ 364us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 21:08:24.493386 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0924 21:08:24.494468 (+ 1082us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 21:08:24.495795 (+ 1327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 21:08:24.495795 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 21:08:24.495796 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 21:08:24.495950 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 21:08:24.496130 (+ 180us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 21:08:24.496131 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 21:08:24.496131 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 21:08:24.496183 (+ 52us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 21:08:24.496414 (+ 231us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 21:08:24.496415 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 21:08:24.496454 (+ 39us) client_negotiation.cc:769] Sending connection context
0924 21:08:24.496522 (+ 68us) client_negotiation.cc:240] Negotiation successful
0924 21:08:24.496607 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28}
I20260924 21:08:24.497138 6476 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.98.60:39715: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "65393c0abc864ec4b9d3d76877a9a863"
member_type: NON_VOTER
last_known_addr {
host: "127.0.98.60"
port: 39715
}
attrs {
promote: true
}
}
dest_uuid: "0f53f4b63d8e4da0905f8db79d1fc7db"
cas_config_opid_index: 5
I20260924 21:08:24.497367 6476 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0f53f4b63d8e4da0905f8db79d1fc7db [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "0f53f4b63d8e4da0905f8db79d1fc7db" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44697 } } peers { permanent_uuid: "a5110fb5b32e4973948078ef72376369" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 34067 } attrs { promote: true } } peers { permanent_uuid: "65393c0abc864ec4b9d3d76877a9a863" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 39715 } attrs { promote: true } }
I20260924 21:08:24.497912 7170 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 65393c0abc864ec4b9d3d76877a9a863 [term 0 FOLLOWER]: Advancing to term 1
I20260924 21:08:24.498817 7170 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 65393c0abc864ec4b9d3d76877a9a863 [term 1 FOLLOWER]: Refusing update from remote peer 0f53f4b63d8e4da0905f8db79d1fc7db: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260924 21:08:24.498971 7036 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 0f53f4b63d8e4da0905f8db79d1fc7db [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 65393c0abc864ec4b9d3d76877a9a863 (127.0.98.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "0f53f4b63d8e4da0905f8db79d1fc7db" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44697 } } peers { permanent_uuid: "a5110fb5b32e4973948078ef72376369" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 34067 } attrs { promote: true } } peers { permanent_uuid: "65393c0abc864ec4b9d3d76877a9a863" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 39715 } attrs { promote: true } } }
I20260924 21:08:24.499853 7036 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.98.60:39715
I20260924 21:08:24.499925 7036 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0f53f4b63d8e4da0905f8db79d1fc7db [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "0f53f4b63d8e4da0905f8db79d1fc7db" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "0f53f4b63d8e4da0905f8db79d1fc7db" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44697 } } peers { permanent_uuid: "a5110fb5b32e4973948078ef72376369" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 34067 } attrs { promote: true } } peers { permanent_uuid: "65393c0abc864ec4b9d3d76877a9a863" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 39715 } attrs { promote: true } } }
I20260924 21:08:24.499991 7036 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0f53f4b63d8e4da0905f8db79d1fc7db [sys.catalog]: This master's current role is: LEADER
I20260924 21:08:24.500172 7128 master.cc:537] Master@127.0.98.60:39715 shutting down...
I20260924 21:08:24.500209 7035 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 0f53f4b63d8e4da0905f8db79d1fc7db [LEADER]: Connected to new peer: Peer: permanent_uuid: "65393c0abc864ec4b9d3d76877a9a863" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 39715 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 21:08:24.500715 7034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:24.497754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:34067 (local address 127.0.0.1:60678)
0924 21:08:24.497845 (+ 91us) negotiation.cc:107] Waiting for socket to connect
0924 21:08:24.497855 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 21:08:24.497898 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 21:08:24.498077 (+ 179us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 21:08:24.498078 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 21:08:24.498079 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 21:08:24.498109 (+ 30us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:24.498110 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:24.498722 (+ 612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:24.498723 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:24.499586 (+ 863us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:24.499589 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:24.500484 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:24.500485 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:24.500545 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:24.500562 (+ 17us) client_negotiation.cc:769] Sending connection context
0924 21:08:24.500619 (+ 57us) client_negotiation.cc:240] Negotiation successful
0924 21:08:24.500650 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":76}
W20260924 21:08:24.500864 6460 consensus_peers.cc:487] T 00000000000000000000000000000000 P 0f53f4b63d8e4da0905f8db79d1fc7db -> Peer 65393c0abc864ec4b9d3d76877a9a863 (127.0.98.60:39715): Couldn't send request to peer 65393c0abc864ec4b9d3d76877a9a863. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260924 21:08:24.500937 7125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:24.497791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60678 (local address 127.0.98.61:34067)
0924 21:08:24.497845 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0924 21:08:24.497847 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 21:08:24.497884 (+ 37us) server_negotiation.cc:407] Connection header received
0924 21:08:24.497942 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:08:24.497942 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:08:24.497962 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:08:24.498034 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 21:08:24.498174 (+ 140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:24.498644 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:24.499676 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:24.500292 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:24.500365 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:24.500452 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0924 21:08:24.500838 (+ 386us) server_negotiation.cc:299] Negotiation successful
0924 21:08:24.500872 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260924 21:08:24.501088 7084 raft_consensus.cc:1261] T 00000000000000000000000000000000 P a5110fb5b32e4973948078ef72376369 [term 1 LEARNER]: Refusing update from remote peer 0f53f4b63d8e4da0905f8db79d1fc7db: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260924 21:08:24.501276 7035 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 0f53f4b63d8e4da0905f8db79d1fc7db [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5110fb5b32e4973948078ef72376369" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 34067 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260924 21:08:24.501756 7084 raft_consensus.cc:2932] T 00000000000000000000000000000000 P a5110fb5b32e4973948078ef72376369 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 65393c0abc864ec4b9d3d76877a9a863 (127.0.98.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "0f53f4b63d8e4da0905f8db79d1fc7db" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44697 } } peers { permanent_uuid: "a5110fb5b32e4973948078ef72376369" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 34067 } attrs { promote: true } } peers { permanent_uuid: "65393c0abc864ec4b9d3d76877a9a863" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 39715 } attrs { promote: true } } }
I20260924 21:08:24.502856 7222 sys_catalog.cc:455] T 00000000000000000000000000000000 P a5110fb5b32e4973948078ef72376369 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0f53f4b63d8e4da0905f8db79d1fc7db" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "0f53f4b63d8e4da0905f8db79d1fc7db" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44697 } } peers { permanent_uuid: "a5110fb5b32e4973948078ef72376369" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 34067 } attrs { promote: true } } peers { permanent_uuid: "65393c0abc864ec4b9d3d76877a9a863" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 39715 } attrs { promote: true } } }
I20260924 21:08:24.502946 7222 sys_catalog.cc:458] T 00000000000000000000000000000000 P a5110fb5b32e4973948078ef72376369 [sys.catalog]: This master's current role is: LEARNER
I20260924 21:08:24.503144 7113 sys_catalog.cc:455] T 00000000000000000000000000000000 P a5110fb5b32e4973948078ef72376369 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0f53f4b63d8e4da0905f8db79d1fc7db. Latest consensus state: current_term: 1 leader_uuid: "0f53f4b63d8e4da0905f8db79d1fc7db" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "0f53f4b63d8e4da0905f8db79d1fc7db" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44697 } } peers { permanent_uuid: "a5110fb5b32e4973948078ef72376369" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 34067 } attrs { promote: true } } peers { permanent_uuid: "65393c0abc864ec4b9d3d76877a9a863" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 39715 } attrs { promote: true } } }
I20260924 21:08:24.503226 7113 sys_catalog.cc:458] T 00000000000000000000000000000000 P a5110fb5b32e4973948078ef72376369 [sys.catalog]: This master's current role is: LEARNER
I20260924 21:08:24.503695 7035 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 0f53f4b63d8e4da0905f8db79d1fc7db: attempting to promote NON_VOTER a5110fb5b32e4973948078ef72376369 to VOTER
I20260924 21:08:24.503875 7035 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0f53f4b63d8e4da0905f8db79d1fc7db [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "0f53f4b63d8e4da0905f8db79d1fc7db" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44697 } } peers { permanent_uuid: "a5110fb5b32e4973948078ef72376369" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34067 } attrs { promote: false } } peers { permanent_uuid: "65393c0abc864ec4b9d3d76877a9a863" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 39715 } attrs { promote: true } }
I20260924 21:08:24.504428 7084 raft_consensus.cc:1261] T 00000000000000000000000000000000 P a5110fb5b32e4973948078ef72376369 [term 1 LEARNER]: Refusing update from remote peer 0f53f4b63d8e4da0905f8db79d1fc7db: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20260924 21:08:24.504631 6460 consensus_peers.cc:487] T 00000000000000000000000000000000 P 0f53f4b63d8e4da0905f8db79d1fc7db -> Peer 65393c0abc864ec4b9d3d76877a9a863 (127.0.98.60:39715): Couldn't send request to peer 65393c0abc864ec4b9d3d76877a9a863. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260924 21:08:24.504729 7223 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 0f53f4b63d8e4da0905f8db79d1fc7db [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5110fb5b32e4973948078ef72376369" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34067 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260924 21:08:24.505275 7128 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 65393c0abc864ec4b9d3d76877a9a863 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 21:08:24.505347 7128 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 65393c0abc864ec4b9d3d76877a9a863 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 21:08:24.505395 7128 tablet_replica.cc:330] T 00000000000000000000000000000000 P 65393c0abc864ec4b9d3d76877a9a863: stopping tablet replica
I20260924 21:08:24.505913 7035 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 0f53f4b63d8e4da0905f8db79d1fc7db [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, a5110fb5b32e4973948078ef72376369 (127.0.98.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "0f53f4b63d8e4da0905f8db79d1fc7db" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44697 } } peers { permanent_uuid: "a5110fb5b32e4973948078ef72376369" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34067 } attrs { promote: false } } peers { permanent_uuid: "65393c0abc864ec4b9d3d76877a9a863" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 39715 } attrs { promote: true } } }
W20260924 21:08:24.506400 6429 connection.cc:535] client connection to 127.0.98.60:39715 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260924 21:08:24.506801 7036 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0f53f4b63d8e4da0905f8db79d1fc7db [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "0f53f4b63d8e4da0905f8db79d1fc7db" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "0f53f4b63d8e4da0905f8db79d1fc7db" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44697 } } peers { permanent_uuid: "a5110fb5b32e4973948078ef72376369" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34067 } attrs { promote: false } } peers { permanent_uuid: "65393c0abc864ec4b9d3d76877a9a863" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 39715 } attrs { promote: true } } }
I20260924 21:08:24.506881 7036 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0f53f4b63d8e4da0905f8db79d1fc7db [sys.catalog]: This master's current role is: LEADER
I20260924 21:08:24.506934 7036 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0f53f4b63d8e4da0905f8db79d1fc7db [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0f53f4b63d8e4da0905f8db79d1fc7db" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "0f53f4b63d8e4da0905f8db79d1fc7db" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44697 } } peers { permanent_uuid: "a5110fb5b32e4973948078ef72376369" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34067 } attrs { promote: false } } peers { permanent_uuid: "65393c0abc864ec4b9d3d76877a9a863" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 39715 } attrs { promote: true } } }
I20260924 21:08:24.506976 7036 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0f53f4b63d8e4da0905f8db79d1fc7db [sys.catalog]: This master's current role is: LEADER
I20260924 21:08:24.507018 7084 raft_consensus.cc:2932] T 00000000000000000000000000000000 P a5110fb5b32e4973948078ef72376369 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, a5110fb5b32e4973948078ef72376369 (127.0.98.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "0f53f4b63d8e4da0905f8db79d1fc7db" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44697 } } peers { permanent_uuid: "a5110fb5b32e4973948078ef72376369" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34067 } attrs { promote: false } } peers { permanent_uuid: "65393c0abc864ec4b9d3d76877a9a863" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 39715 } attrs { promote: true } } }
I20260924 21:08:24.507838 7113 sys_catalog.cc:455] T 00000000000000000000000000000000 P a5110fb5b32e4973948078ef72376369 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0f53f4b63d8e4da0905f8db79d1fc7db" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "0f53f4b63d8e4da0905f8db79d1fc7db" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44697 } } peers { permanent_uuid: "a5110fb5b32e4973948078ef72376369" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34067 } attrs { promote: false } } peers { permanent_uuid: "65393c0abc864ec4b9d3d76877a9a863" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 39715 } attrs { promote: true } } }
I20260924 21:08:24.507925 7113 sys_catalog.cc:458] T 00000000000000000000000000000000 P a5110fb5b32e4973948078ef72376369 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 21:08:24.518429 7128 master.cc:552] Master@127.0.98.60:39715 shutdown complete.
I20260924 21:08:24.518489 7128 master_runner.cc:305] Clearing existing system tablet
I20260924 21:08:24.518649 7128 fs_manager.cc:359] Metadata directory not provided
I20260924 21:08:24.518711 7128 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 21:08:24.519331 7128 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.000s sys 0.001s
I20260924 21:08:24.519847 7226 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 21:08:24.519966 7128 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790284076800039-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 21:08:24.520022 7128 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 21:08:24.520048 7128 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790284076800039-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790284076800039-392-0/minicluster-data/master-2/wal
uuid: "65393c0abc864ec4b9d3d76877a9a863"
format_stamp: "Formatted at 2026-09-24 21:08:24 on dist-test-slave-c7tk"
server_key: "929801a3804c76335e16b5f9c98a0a5a"
server_key_iv: "8b4f4eb4f212052792984ddcb255de39"
server_key_version: "encryptionkey@0"
I20260924 21:08:24.520586 7128 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 65393c0abc864ec4b9d3d76877a9a863: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 21:08:24.521905 7128 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 65393c0abc864ec4b9d3d76877a9a863: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260924 21:08:24.521955 7128 log.cc:1196] T 00000000000000000000000000000000 P 65393c0abc864ec4b9d3d76877a9a863: Deleting WAL directory at /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790284076800039-392-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260924 21:08:24.522122 7128 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 65393c0abc864ec4b9d3d76877a9a863: Deleting consensus metadata
I20260924 21:08:24.522228 7128 master_runner.cc:315] Copying system tablet from 127.0.98.62:44697
I20260924 21:08:24.522812 7128 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 65393c0abc864ec4b9d3d76877a9a863: tablet copy: Beginning tablet copy session from remote peer at address 127.0.98.62:44697
I20260924 21:08:24.528232 6516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:24.523139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33652 (local address 127.0.98.62:44697)
0924 21:08:24.523178 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0924 21:08:24.523180 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 21:08:24.523233 (+ 53us) server_negotiation.cc:407] Connection header received
0924 21:08:24.523306 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:08:24.523306 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:08:24.523315 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:08:24.523365 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:08:24.523640 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:24.524038 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:24.524698 (+ 660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:24.524879 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:24.524952 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:24.525824 (+ 872us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 21:08:24.525826 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 21:08:24.525826 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 21:08:24.525833 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 21:08:24.527140 (+ 1307us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:08:24.527459 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:08:24.527460 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:08:24.527460 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:08:24.527559 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:08:24.527754 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:08:24.527755 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:08:24.527755 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:08:24.527861 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 21:08:24.527935 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0924 21:08:24.528079 (+ 144us) server_negotiation.cc:299] Negotiation successful
0924 21:08:24.528150 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260924 21:08:24.528244 7232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:24.523085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44697 (local address 127.0.0.1:33652)
0924 21:08:24.523180 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0924 21:08:24.523189 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 21:08:24.523247 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 21:08:24.523370 (+ 123us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 21:08:24.523371 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 21:08:24.523531 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0924 21:08:24.523573 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:24.523574 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:24.524107 (+ 533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:24.524108 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:24.524607 (+ 499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:24.524609 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:24.524954 (+ 345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:24.524954 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:24.525014 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:24.525454 (+ 440us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 21:08:24.525458 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 21:08:24.525701 (+ 243us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 21:08:24.527222 (+ 1521us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 21:08:24.527222 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 21:08:24.527223 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 21:08:24.527378 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 21:08:24.527624 (+ 246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 21:08:24.527625 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 21:08:24.527625 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 21:08:24.527685 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 21:08:24.527927 (+ 242us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 21:08:24.527928 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 21:08:24.527969 (+ 41us) client_negotiation.cc:769] Sending connection context
0924 21:08:24.528056 (+ 87us) client_negotiation.cc:240] Negotiation successful
0924 21:08:24.528150 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":73,"thread_start_us":57,"threads_started":1}
I20260924 21:08:24.528684 6496 tablet_copy_service.cc:140] P 0f53f4b63d8e4da0905f8db79d1fc7db: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 65393c0abc864ec4b9d3d76877a9a863 ({username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:33652)
I20260924 21:08:24.528748 6496 tablet_copy_service.cc:161] P 0f53f4b63d8e4da0905f8db79d1fc7db: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 65393c0abc864ec4b9d3d76877a9a863 at {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:33652: session id = 65393c0abc864ec4b9d3d76877a9a863-00000000000000000000000000000000
I20260924 21:08:24.529268 6496 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 0f53f4b63d8e4da0905f8db79d1fc7db: Tablet Copy: opened 0 blocks and 1 log segments
I20260924 21:08:24.529685 7128 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260924 21:08:24.530508 7128 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 65393c0abc864ec4b9d3d76877a9a863: tablet copy: Starting download of 0 data blocks...
I20260924 21:08:24.530653 7128 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 65393c0abc864ec4b9d3d76877a9a863: tablet copy: Starting download of 1 WAL segments...
I20260924 21:08:24.531364 7128 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 65393c0abc864ec4b9d3d76877a9a863: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260924 21:08:24.532104 6496 tablet_copy_service.cc:342] P 0f53f4b63d8e4da0905f8db79d1fc7db: Request end of tablet copy session 65393c0abc864ec4b9d3d76877a9a863-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:33652
I20260924 21:08:24.532152 6496 tablet_copy_service.cc:433] P 0f53f4b63d8e4da0905f8db79d1fc7db: ending tablet copy session 65393c0abc864ec4b9d3d76877a9a863-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 65393c0abc864ec4b9d3d76877a9a863
I20260924 21:08:24.534164 7128 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 21:08:24.535274 7236 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 21:08:24.535413 7233 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 21:08:24.535439 7234 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 21:08:24.535580 7128 server_base.cc:1034] running on GCE node
I20260924 21:08:24.535688 7128 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 21:08:24.535808 7128 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 21:08:24.536921 7128 hybrid_clock.cc:648] HybridClock initialized: now 1790284104536916 us; error 20 us; skew 500 ppm
Sep 24 21:08:24 dist-test-slave-c7tk krb5kdc[6431](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284104, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 21:08:24.537963 7128 init.cc:377] Logged in from keytab as oryx/127.0.98.60@KRBTEST.COM (short username oryx)
I20260924 21:08:24.538195 7128 webserver.cc:460] Webserver started at http://127.0.98.60:45833/ using document root <none> and password file <none>
I20260924 21:08:24.538293 7128 fs_manager.cc:359] Metadata directory not provided
I20260924 21:08:24.538323 7128 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 21:08:24.538880 7128 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.000s sys 0.000s
I20260924 21:08:24.539373 7243 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 21:08:24.539512 7128 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 21:08:24.539558 7128 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790284076800039-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790284076800039-392-0/minicluster-data/master-2/wal
uuid: "65393c0abc864ec4b9d3d76877a9a863"
format_stamp: "Formatted at 2026-09-24 21:08:24 on dist-test-slave-c7tk"
server_key: "929801a3804c76335e16b5f9c98a0a5a"
server_key_iv: "8b4f4eb4f212052792984ddcb255de39"
server_key_version: "encryptionkey@0"
I20260924 21:08:24.539608 7128 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790284076800039-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790284076800039-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790284076800039-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 21:08:24.555392 7128 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 21:08:24.556829 7128 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 21:08:24.560011 7128 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:39715
I20260924 21:08:24.560209 7295 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:39715 every 8 connection(s)
I20260924 21:08:24.560293 7128 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790284076800039-392-0/minicluster-data/master-2/data/info.pb
I20260924 21:08:24.560971 7296 sys_catalog.cc:263] Verifying existing consensus state
I20260924 21:08:24.561477 7296 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 65393c0abc864ec4b9d3d76877a9a863: Bootstrap starting.
I20260924 21:08:24.564038 7296 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 65393c0abc864ec4b9d3d76877a9a863: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 21:08:24.564235 7296 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 65393c0abc864ec4b9d3d76877a9a863: Bootstrap complete.
I20260924 21:08:24.564405 7296 raft_consensus.cc:348] T 00000000000000000000000000000000 P 65393c0abc864ec4b9d3d76877a9a863 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "0f53f4b63d8e4da0905f8db79d1fc7db" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44697 } } peers { permanent_uuid: "a5110fb5b32e4973948078ef72376369" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34067 } attrs { promote: false } } peers { permanent_uuid: "65393c0abc864ec4b9d3d76877a9a863" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 39715 } attrs { promote: true } }
I20260924 21:08:24.564483 7296 raft_consensus.cc:729] T 00000000000000000000000000000000 P 65393c0abc864ec4b9d3d76877a9a863 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 65393c0abc864ec4b9d3d76877a9a863, State: Initialized, Role: LEARNER
I20260924 21:08:24.564514 7296 consensus_queue.cc:260] T 00000000000000000000000000000000 P 65393c0abc864ec4b9d3d76877a9a863 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "0f53f4b63d8e4da0905f8db79d1fc7db" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44697 } } peers { permanent_uuid: "a5110fb5b32e4973948078ef72376369" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34067 } attrs { promote: false } } peers { permanent_uuid: "65393c0abc864ec4b9d3d76877a9a863" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 39715 } attrs { promote: true } }
I20260924 21:08:24.564647 7296 sys_catalog.cc:564] T 00000000000000000000000000000000 P 65393c0abc864ec4b9d3d76877a9a863 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 21:08:24.564656 7299 sys_catalog.cc:455] T 00000000000000000000000000000000 P 65393c0abc864ec4b9d3d76877a9a863 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "0f53f4b63d8e4da0905f8db79d1fc7db" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44697 } } peers { permanent_uuid: "a5110fb5b32e4973948078ef72376369" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34067 } attrs { promote: false } } peers { permanent_uuid: "65393c0abc864ec4b9d3d76877a9a863" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 39715 } attrs { promote: true } } }
I20260924 21:08:24.564774 7299 sys_catalog.cc:458] T 00000000000000000000000000000000 P 65393c0abc864ec4b9d3d76877a9a863 [sys.catalog]: This master's current role is: LEARNER
I20260924 21:08:24.566004 7310 catalog_manager.cc:1259] Loaded cluster ID: e2489afc08c64420b12bd07cdab5f107
I20260924 21:08:24.566468 7310 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 65393c0abc864ec4b9d3d76877a9a863: loading cluster ID for follower catalog manager: success
I20260924 21:08:24.567075 7310 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 65393c0abc864ec4b9d3d76877a9a863: acquiring CA information for follower catalog manager: success
I20260924 21:08:24.567302 7310 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 65393c0abc864ec4b9d3d76877a9a863: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 21:08:24.612258 7311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:24.607343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57300 (local address 127.0.98.60:39715)
0924 21:08:24.607446 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0924 21:08:24.607448 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 21:08:24.607455 (+ 7us) server_negotiation.cc:407] Connection header received
0924 21:08:24.607474 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:08:24.607474 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:08:24.607482 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:08:24.607533 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:08:24.607978 (+ 445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:24.608500 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:24.609167 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:24.609341 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:24.609388 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:24.610483 (+ 1095us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 21:08:24.610484 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 21:08:24.610484 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 21:08:24.610488 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 21:08:24.611503 (+ 1015us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:08:24.611763 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:08:24.611764 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:08:24.611764 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:08:24.611804 (+ 40us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:08:24.611936 (+ 132us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:08:24.611936 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:08:24.611937 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:08:24.612007 (+ 70us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 21:08:24.612066 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0924 21:08:24.612132 (+ 66us) server_negotiation.cc:299] Negotiation successful
0924 21:08:24.612178 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88,"thread_start_us":57,"threads_started":1}
I20260924 21:08:24.612710 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAzNNjS/build/release/bin/kudu with pid 6538
I20260924 21:08:24.616999 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAzNNjS/build/release/bin/kudu
/tmp/dist-test-taskAzNNjS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790284076800039-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790284076800039-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790284076800039-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790284076800039-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:41965
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:44697
--webserver_port=39407
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790284076800039-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37913
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:44697,127.0.98.61:34067
--tserver_master_addrs=127.0.98.62:44697,127.0.98.61:34067,127.0.98.60:39715 with env {KRB5CCNAME=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790284076800039-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790284076800039-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790284076800039-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 21:08:24.689932 7312 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 21:08:24.690127 7312 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 21:08:24.690151 7312 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 21:08:24.691288 7312 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 21:08:24.691332 7312 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 21:08:24.691355 7312 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 21:08:24.692505 7312 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37913
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790284076800039-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790284076800039-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790284076800039-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:41965
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790284076800039-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=39407
--tserver_master_addrs=127.0.98.62:44697,127.0.98.61:34067,127.0.98.60:39715
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7312
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790284076800039-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 19:49:06 UTC on ccycloud.quasar-vhqjoc.root.comops.site
build id 4670864
I20260924 21:08:24.692653 7312 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 21:08:24.692786 7312 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 21:08:24.696979 7320 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 21:08:24.696990 7318 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 21:08:24.697137 7312 server_base.cc:1034] running on GCE node
W20260924 21:08:24.696995 7317 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 21:08:24.697412 7312 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 21:08:24.697683 7312 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 21:08:24.698824 7312 hybrid_clock.cc:648] HybridClock initialized: now 1790284104698821 us; error 38 us; skew 500 ppm
Sep 24 21:08:24 dist-test-slave-c7tk krb5kdc[6431](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284104, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 21:08:24.701056 7312 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260924 21:08:24.701372 7312 webserver.cc:460] Webserver started at http://127.0.98.1:39407/ using document root <none> and password file <none>
I20260924 21:08:24.701520 7312 fs_manager.cc:359] Metadata directory not provided
I20260924 21:08:24.701561 7312 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 21:08:24.702591 7312 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.000s sys 0.000s
I20260924 21:08:24.703032 7327 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 21:08:24.703177 7312 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 21:08:24.703248 7312 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790284076800039-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790284076800039-392-0/minicluster-data/ts-0/wal
uuid: "1111a5950630489ba30bd8b925f7a15b"
format_stamp: "Formatted at 2026-09-24 21:08:23 on dist-test-slave-c7tk"
server_key: "7b301a44a3bb1dcac9b0c435adc707b8"
server_key_iv: "548b23a9211ad056ee847c20876171f9"
server_key_version: "encryptionkey@0"
I20260924 21:08:24.703464 7312 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790284076800039-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790284076800039-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790284076800039-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 21:08:24.714274 7312 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 21:08:24.716001 7312 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 21:08:24.716089 7312 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 21:08:24.716213 7312 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 21:08:24.716393 7312 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 21:08:24.716418 7312 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 21:08:24.716434 7312 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 21:08:24.716444 7312 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 21:08:24.721478 7312 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:41965
I20260924 21:08:24.721524 7440 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:41965 every 8 connection(s)
I20260924 21:08:24.721843 7312 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790284076800039-392-0/minicluster-data/ts-0/data/info.pb
Sep 24 21:08:24 dist-test-slave-c7tk krb5kdc[6431](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284104, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
Sep 24 21:08:24 dist-test-slave-c7tk krb5kdc[6431](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284104, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
Sep 24 21:08:24 dist-test-slave-c7tk krb5kdc[6431](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284104, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260924 21:08:24.730902 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAzNNjS/build/release/bin/kudu as pid 7312
I20260924 21:08:24.731050 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAzNNjS/build/release/bin/kudu with pid 6672
I20260924 21:08:24.732666 6516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:24.722851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:37881 (local address 127.0.98.62:44697)
0924 21:08:24.722910 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0924 21:08:24.722912 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 21:08:24.722973 (+ 61us) server_negotiation.cc:407] Connection header received
0924 21:08:24.724041 (+ 1068us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:08:24.724042 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:08:24.724054 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:08:24.724082 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:08:24.724827 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:24.725567 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:24.727144 (+ 1577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:24.727425 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:24.727498 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:24.729984 (+ 2486us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 21:08:24.729986 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 21:08:24.729986 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 21:08:24.729992 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 21:08:24.731251 (+ 1259us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:08:24.731869 (+ 618us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:08:24.731870 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:08:24.731870 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:08:24.731917 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:08:24.732135 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:08:24.732136 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:08:24.732136 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:08:24.732237 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 21:08:24.732303 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0924 21:08:24.732521 (+ 218us) server_negotiation.cc:299] Negotiation successful
0924 21:08:24.732589 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260924 21:08:24.732869 7447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:24.722807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44697 (local address 127.0.98.1:37881)
0924 21:08:24.722917 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0924 21:08:24.722927 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 21:08:24.723946 (+ 1019us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 21:08:24.724154 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 21:08:24.724155 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 21:08:24.724552 (+ 397us) client_negotiation.cc:189] Negotiated authn=SASL
0924 21:08:24.724731 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:24.724737 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:24.725640 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:24.725641 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:24.726588 (+ 947us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:24.726591 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:24.727500 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:24.727501 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:24.727578 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:24.727937 (+ 359us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 21:08:24.727942 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 21:08:24.729913 (+ 1971us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 21:08:24.731628 (+ 1715us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 21:08:24.731629 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 21:08:24.731629 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 21:08:24.731794 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 21:08:24.732004 (+ 210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 21:08:24.732005 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 21:08:24.732005 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 21:08:24.732079 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 21:08:24.732294 (+ 215us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 21:08:24.732299 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 21:08:24.732416 (+ 117us) client_negotiation.cc:769] Sending connection context
0924 21:08:24.732503 (+ 87us) client_negotiation.cc:240] Negotiation successful
0924 21:08:24.732628 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":96,"mutex_wait_us":43,"thread_start_us":78,"threads_started":1}
I20260924 21:08:24.733204 7442 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44697
I20260924 21:08:24.733275 7442 heartbeater.cc:464] Registering TS with master...
I20260924 21:08:24.733428 7442 heartbeater.cc:511] Master 127.0.98.62:44697 requested a full tablet report, sending...
I20260924 21:08:24.733546 7311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:24.722526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:36357 (local address 127.0.98.60:39715)
0924 21:08:24.722574 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0924 21:08:24.722577 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 21:08:24.722745 (+ 168us) server_negotiation.cc:407] Connection header received
0924 21:08:24.724042 (+ 1297us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:08:24.724044 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:08:24.724055 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:08:24.724082 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:08:24.724834 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:24.725555 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:24.726715 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:24.727042 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:24.727101 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:24.729423 (+ 2322us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 21:08:24.729424 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 21:08:24.729424 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 21:08:24.729431 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 21:08:24.732591 (+ 3160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:08:24.732920 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:08:24.732921 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:08:24.732922 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:08:24.732971 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:08:24.733116 (+ 145us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:08:24.733117 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:08:24.733117 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:08:24.733209 (+ 92us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 21:08:24.733240 (+ 31us) server_negotiation.cc:1035] Waiting for connection context
0924 21:08:24.733391 (+ 151us) server_negotiation.cc:299] Negotiation successful
0924 21:08:24.733460 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260924 21:08:24.733889 6476 ts_manager.cc:194] Re-registered known tserver with Master: 1111a5950630489ba30bd8b925f7a15b (127.0.98.1:41965)
I20260924 21:08:24.733944 7445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:24.722496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39715 (local address 127.0.98.1:36357)
0924 21:08:24.722644 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0924 21:08:24.722681 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0924 21:08:24.723938 (+ 1257us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 21:08:24.724139 (+ 201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 21:08:24.724140 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 21:08:24.724548 (+ 408us) client_negotiation.cc:189] Negotiated authn=SASL
0924 21:08:24.724731 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:24.724737 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:24.725640 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:24.725641 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:24.726588 (+ 947us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:24.726591 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:24.727112 (+ 521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:24.727113 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:24.727192 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:24.727739 (+ 547us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 21:08:24.727743 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 21:08:24.729306 (+ 1563us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 21:08:24.732666 (+ 3360us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 21:08:24.732667 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 21:08:24.732667 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 21:08:24.732846 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 21:08:24.733026 (+ 180us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 21:08:24.733027 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 21:08:24.733027 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 21:08:24.733067 (+ 40us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 21:08:24.733273 (+ 206us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 21:08:24.733274 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 21:08:24.733305 (+ 31us) client_negotiation.cc:769] Sending connection context
0924 21:08:24.733631 (+ 326us) client_negotiation.cc:240] Negotiation successful
0924 21:08:24.733714 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"mutex_wait_us":136,"thread_start_us":95,"threads_started":1}
I20260924 21:08:24.734270 6476 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:37881
I20260924 21:08:24.734886 7441 heartbeater.cc:347] Connected to a master server at 127.0.98.60:39715
I20260924 21:08:24.734930 7441 heartbeater.cc:464] Registering TS with master...
I20260924 21:08:24.734987 7441 heartbeater.cc:511] Master 127.0.98.60:39715 requested a full tablet report, sending...
I20260924 21:08:24.735368 7260 ts_manager.cc:194] Registered new tserver with Master: 1111a5950630489ba30bd8b925f7a15b (127.0.98.1:41965)
I20260924 21:08:24.736369 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAzNNjS/build/release/bin/kudu
/tmp/dist-test-taskAzNNjS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790284076800039-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790284076800039-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790284076800039-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790284076800039-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:46665
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:44697
--webserver_port=37657
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790284076800039-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37913
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:44697,127.0.98.61:34067
--tserver_master_addrs=127.0.98.62:44697,127.0.98.61:34067,127.0.98.60:39715 with env {KRB5CCNAME=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790284076800039-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790284076800039-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790284076800039-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 21:08:24.736778 7125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:24.722587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:43401 (local address 127.0.98.61:34067)
0924 21:08:24.722630 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0924 21:08:24.722632 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 21:08:24.722756 (+ 124us) server_negotiation.cc:407] Connection header received
0924 21:08:24.724026 (+ 1270us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:08:24.724027 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:08:24.724037 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:08:24.724080 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:08:24.724813 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:24.725258 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:24.726286 (+ 1028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:24.726484 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:24.726564 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:24.728919 (+ 2355us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 21:08:24.728921 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 21:08:24.728921 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 21:08:24.728927 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 21:08:24.730430 (+ 1503us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:08:24.730867 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:08:24.730868 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:08:24.730869 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:08:24.730943 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:08:24.736256 (+ 5313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:08:24.736257 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:08:24.736257 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:08:24.736355 (+ 98us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 21:08:24.736413 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0924 21:08:24.736581 (+ 168us) server_negotiation.cc:299] Negotiation successful
0924 21:08:24.736656 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260924 21:08:24.737100 7446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:24.722496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:34067 (local address 127.0.98.1:43401)
0924 21:08:24.722682 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0924 21:08:24.722692 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 21:08:24.723939 (+ 1247us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 21:08:24.724131 (+ 192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 21:08:24.724134 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 21:08:24.724533 (+ 399us) client_negotiation.cc:189] Negotiated authn=SASL
0924 21:08:24.724730 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:24.724733 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:24.725385 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:24.725385 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:24.726192 (+ 807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:24.726194 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:24.726572 (+ 378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:24.726574 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:24.726713 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:24.727141 (+ 428us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 21:08:24.727149 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 21:08:24.728837 (+ 1688us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 21:08:24.730505 (+ 1668us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 21:08:24.730510 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 21:08:24.730512 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 21:08:24.730771 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 21:08:24.731022 (+ 251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 21:08:24.731023 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 21:08:24.731023 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 21:08:24.736188 (+ 5165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 21:08:24.736427 (+ 239us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 21:08:24.736427 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 21:08:24.736468 (+ 41us) client_negotiation.cc:769] Sending connection context
0924 21:08:24.736857 (+ 389us) client_negotiation.cc:240] Negotiation successful
0924 21:08:24.736958 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156,"mutex_wait_us":1,"thread_start_us":87,"threads_started":1}
I20260924 21:08:24.737416 7443 heartbeater.cc:347] Connected to a master server at 127.0.98.61:34067
I20260924 21:08:24.737498 7443 heartbeater.cc:464] Registering TS with master...
I20260924 21:08:24.737546 7443 heartbeater.cc:511] Master 127.0.98.61:34067 requested a full tablet report, sending...
I20260924 21:08:24.737890 7074 ts_manager.cc:194] Registered new tserver with Master: 1111a5950630489ba30bd8b925f7a15b (127.0.98.1:41965)
W20260924 21:08:24.809938 7448 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 21:08:24.810113 7448 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 21:08:24.810148 7448 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 21:08:24.811201 7448 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 21:08:24.811255 7448 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 21:08:24.811293 7448 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 21:08:24.812492 7448 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37913
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790284076800039-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790284076800039-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790284076800039-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:46665
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790284076800039-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=37657
--tserver_master_addrs=127.0.98.62:44697,127.0.98.61:34067,127.0.98.60:39715
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7448
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790284076800039-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 19:49:06 UTC on ccycloud.quasar-vhqjoc.root.comops.site
build id 4670864
I20260924 21:08:24.812712 7448 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 21:08:24.812899 7448 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 21:08:24.817225 7453 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 21:08:24.817237 7454 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 21:08:24.817341 7448 server_base.cc:1034] running on GCE node
W20260924 21:08:24.817237 7456 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 21:08:24.817586 7448 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 21:08:24.817786 7448 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 21:08:24.818954 7448 hybrid_clock.cc:648] HybridClock initialized: now 1790284104818942 us; error 25 us; skew 500 ppm
Sep 24 21:08:24 dist-test-slave-c7tk krb5kdc[6431](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284104, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 21:08:24.821133 7448 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260924 21:08:24.821463 7448 webserver.cc:460] Webserver started at http://127.0.98.2:37657/ using document root <none> and password file <none>
I20260924 21:08:24.821652 7448 fs_manager.cc:359] Metadata directory not provided
I20260924 21:08:24.821722 7448 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 21:08:24.822778 7448 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 21:08:24.823211 7463 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 21:08:24.823339 7448 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 21:08:24.823418 7448 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790284076800039-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790284076800039-392-0/minicluster-data/ts-1/wal
uuid: "80a0113d8614402db4980fda67be9b4f"
format_stamp: "Formatted at 2026-09-24 21:08:23 on dist-test-slave-c7tk"
server_key: "4cead28aafe68f4b0c1e5bb4ef186ccc"
server_key_iv: "5cfc7a0724ce40c647813a9d18503e34"
server_key_version: "encryptionkey@0"
I20260924 21:08:24.823664 7448 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790284076800039-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790284076800039-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790284076800039-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 21:08:24.841754 7448 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 21:08:24.843325 7448 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 21:08:24.843458 7448 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 21:08:24.843653 7448 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 21:08:24.843927 7448 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 21:08:24.843966 7448 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 21:08:24.843997 7448 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 21:08:24.844014 7448 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 21:08:24.848949 7448 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:46665
I20260924 21:08:24.848981 7576 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:46665 every 8 connection(s)
I20260924 21:08:24.849275 7448 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790284076800039-392-0/minicluster-data/ts-1/data/info.pb
I20260924 21:08:24.850652 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAzNNjS/build/release/bin/kudu as pid 7448
I20260924 21:08:24.850827 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAzNNjS/build/release/bin/kudu with pid 6806
Sep 24 21:08:24 dist-test-slave-c7tk krb5kdc[6431](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284104, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
Sep 24 21:08:24 dist-test-slave-c7tk krb5kdc[6431](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284104, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260924 21:08:24.857632 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAzNNjS/build/release/bin/kudu
/tmp/dist-test-taskAzNNjS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790284076800039-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790284076800039-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790284076800039-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790284076800039-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:37191
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:44697
--webserver_port=38997
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790284076800039-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37913
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:44697,127.0.98.61:34067
--tserver_master_addrs=127.0.98.62:44697,127.0.98.61:34067,127.0.98.60:39715 with env {KRB5CCNAME=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790284076800039-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790284076800039-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790284076800039-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 21:08:24 dist-test-slave-c7tk krb5kdc[6431](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284104, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260924 21:08:24.860566 7311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:24.850158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:44901 (local address 127.0.98.60:39715)
0924 21:08:24.850227 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0924 21:08:24.850229 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 21:08:24.850238 (+ 9us) server_negotiation.cc:407] Connection header received
0924 21:08:24.851759 (+ 1521us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:08:24.851761 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:08:24.851772 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:08:24.851818 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:08:24.852280 (+ 462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:24.852933 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:24.854351 (+ 1418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:24.854609 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:24.854647 (+ 38us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:24.857090 (+ 2443us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 21:08:24.857091 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 21:08:24.857092 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 21:08:24.857097 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 21:08:24.859068 (+ 1971us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:08:24.859649 (+ 581us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:08:24.859650 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:08:24.859650 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:08:24.859707 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:08:24.859975 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:08:24.859976 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:08:24.859976 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:08:24.860085 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 21:08:24.860144 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0924 21:08:24.860422 (+ 278us) server_negotiation.cc:299] Negotiation successful
0924 21:08:24.860492 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260924 21:08:24.861182 6516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:24.849931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:40585 (local address 127.0.98.62:44697)
0924 21:08:24.849973 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0924 21:08:24.849975 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 21:08:24.850164 (+ 189us) server_negotiation.cc:407] Connection header received
0924 21:08:24.851259 (+ 1095us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:08:24.851260 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:08:24.851273 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:08:24.851302 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:08:24.852131 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:24.852845 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:24.854348 (+ 1503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:24.854673 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:24.854714 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:24.857482 (+ 2768us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 21:08:24.857483 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 21:08:24.857483 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 21:08:24.857489 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 21:08:24.859052 (+ 1563us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:08:24.859719 (+ 667us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:08:24.859720 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:08:24.859720 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:08:24.859812 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:08:24.860044 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:08:24.860045 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:08:24.860045 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:08:24.860155 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 21:08:24.860202 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0924 21:08:24.860992 (+ 790us) server_negotiation.cc:299] Negotiation successful
0924 21:08:24.861066 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 21:08:24.861654 7582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:24.849938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44697 (local address 127.0.98.2:40585)
0924 21:08:24.850072 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0924 21:08:24.850103 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0924 21:08:24.851156 (+ 1053us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 21:08:24.851457 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 21:08:24.851461 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 21:08:24.851802 (+ 341us) client_negotiation.cc:189] Negotiated authn=SASL
0924 21:08:24.852036 (+ 234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:24.852041 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:24.852951 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:24.852952 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:24.854259 (+ 1307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:24.854261 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:24.854790 (+ 529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:24.854791 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:24.854866 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:24.855497 (+ 631us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 21:08:24.855503 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0924 21:08:24.857382 (+ 1879us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 21:08:24.859145 (+ 1763us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 21:08:24.859153 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 21:08:24.859155 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 21:08:24.859612 (+ 457us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 21:08:24.859917 (+ 305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 21:08:24.859918 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 21:08:24.859918 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 21:08:24.859972 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 21:08:24.860805 (+ 833us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 21:08:24.860806 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 21:08:24.860846 (+ 40us) client_negotiation.cc:769] Sending connection context
0924 21:08:24.861381 (+ 535us) client_negotiation.cc:240] Negotiation successful
0924 21:08:24.861480 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103,"mutex_wait_us":86,"thread_start_us":69,"threads_started":1}
I20260924 21:08:24.861918 7581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:24.849939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39715 (local address 127.0.98.2:44901)
0924 21:08:24.850074 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0924 21:08:24.850104 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0924 21:08:24.851687 (+ 1583us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 21:08:24.851870 (+ 183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 21:08:24.851872 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 21:08:24.852100 (+ 228us) client_negotiation.cc:189] Negotiated authn=SASL
0924 21:08:24.852193 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:24.852194 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:24.853008 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:24.853009 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:24.854258 (+ 1249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:24.854260 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:24.854728 (+ 468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:24.854729 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:24.854863 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:24.855598 (+ 735us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 21:08:24.855601 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0924 21:08:24.856997 (+ 1396us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 21:08:24.859145 (+ 2148us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 21:08:24.859153 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 21:08:24.859155 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 21:08:24.859569 (+ 414us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 21:08:24.859793 (+ 224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 21:08:24.859794 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 21:08:24.859794 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 21:08:24.859929 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 21:08:24.860150 (+ 221us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 21:08:24.860156 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 21:08:24.860286 (+ 130us) client_negotiation.cc:769] Sending connection context
0924 21:08:24.860393 (+ 107us) client_negotiation.cc:240] Negotiation successful
0924 21:08:24.860532 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"mutex_wait_us":110,"thread_start_us":78,"threads_started":1}
I20260924 21:08:24.862098 7578 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44697
I20260924 21:08:24.862169 7578 heartbeater.cc:464] Registering TS with master...
I20260924 21:08:24.862227 7577 heartbeater.cc:347] Connected to a master server at 127.0.98.60:39715
I20260924 21:08:24.862272 7577 heartbeater.cc:464] Registering TS with master...
I20260924 21:08:24.862332 7578 heartbeater.cc:511] Master 127.0.98.62:44697 requested a full tablet report, sending...
I20260924 21:08:24.862336 7577 heartbeater.cc:511] Master 127.0.98.60:39715 requested a full tablet report, sending...
I20260924 21:08:24.862488 7583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:24.850181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:34067 (local address 127.0.98.2:56659)
0924 21:08:24.850285 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0924 21:08:24.850293 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0924 21:08:24.851760 (+ 1467us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 21:08:24.851930 (+ 170us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 21:08:24.851930 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 21:08:24.852112 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0924 21:08:24.852194 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:24.852194 (+ 0us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:24.853008 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:24.853009 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:24.854258 (+ 1249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:24.854260 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:24.854727 (+ 467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:24.854728 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:24.854853 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:24.855438 (+ 585us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 21:08:24.855447 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 21:08:24.859538 (+ 4091us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 21:08:24.861256 (+ 1718us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 21:08:24.861257 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 21:08:24.861257 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 21:08:24.861436 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 21:08:24.861623 (+ 187us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 21:08:24.861624 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 21:08:24.861624 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 21:08:24.861671 (+ 47us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 21:08:24.862140 (+ 469us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 21:08:24.862141 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 21:08:24.862234 (+ 93us) client_negotiation.cc:769] Sending connection context
0924 21:08:24.862300 (+ 66us) client_negotiation.cc:240] Negotiation successful
0924 21:08:24.862388 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":90,"mutex_wait_us":117,"thread_start_us":61,"threads_started":1}
I20260924 21:08:24.862521 7125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:24.850277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:56659 (local address 127.0.98.61:34067)
0924 21:08:24.850310 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0924 21:08:24.850312 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 21:08:24.850323 (+ 11us) server_negotiation.cc:407] Connection header received
0924 21:08:24.851834 (+ 1511us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:08:24.851835 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:08:24.851846 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:08:24.851874 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:08:24.852281 (+ 407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:24.852921 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:24.854363 (+ 1442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:24.854624 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:24.854656 (+ 32us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:24.859638 (+ 4982us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 21:08:24.859639 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 21:08:24.859640 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 21:08:24.859646 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 21:08:24.861176 (+ 1530us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:08:24.861510 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:08:24.861511 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:08:24.861511 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:08:24.861564 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:08:24.861760 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:08:24.861761 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:08:24.861762 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:08:24.861914 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 21:08:24.861971 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0924 21:08:24.862339 (+ 368us) server_negotiation.cc:299] Negotiation successful
0924 21:08:24.862416 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260924 21:08:24.862762 6476 ts_manager.cc:194] Re-registered known tserver with Master: 80a0113d8614402db4980fda67be9b4f (127.0.98.2:46665)
I20260924 21:08:24.862815 7260 ts_manager.cc:194] Registered new tserver with Master: 80a0113d8614402db4980fda67be9b4f (127.0.98.2:46665)
I20260924 21:08:24.863032 7579 heartbeater.cc:347] Connected to a master server at 127.0.98.61:34067
I20260924 21:08:24.863070 7579 heartbeater.cc:464] Registering TS with master...
I20260924 21:08:24.863121 7579 heartbeater.cc:511] Master 127.0.98.61:34067 requested a full tablet report, sending...
I20260924 21:08:24.863245 6476 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:40585
I20260924 21:08:24.863361 7074 ts_manager.cc:194] Registered new tserver with Master: 80a0113d8614402db4980fda67be9b4f (127.0.98.2:46665)
W20260924 21:08:24.930672 7584 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 21:08:24.930819 7584 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 21:08:24.930836 7584 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 21:08:24.931771 7584 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 21:08:24.931804 7584 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 21:08:24.931825 7584 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 21:08:24.933046 7584 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37913
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790284076800039-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790284076800039-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790284076800039-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:37191
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790284076800039-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=38997
--tserver_master_addrs=127.0.98.62:44697,127.0.98.61:34067,127.0.98.60:39715
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7584
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790284076800039-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 19:49:06 UTC on ccycloud.quasar-vhqjoc.root.comops.site
build id 4670864
I20260924 21:08:24.933192 7584 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 21:08:24.933322 7584 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 21:08:24.937469 7592 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 21:08:24.937512 7590 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 21:08:24.937593 7584 server_base.cc:1034] running on GCE node
W20260924 21:08:24.937476 7589 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 21:08:24.937896 7584 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 21:08:24.938105 7584 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 21:08:24.939230 7584 hybrid_clock.cc:648] HybridClock initialized: now 1790284104939206 us; error 38 us; skew 500 ppm
Sep 24 21:08:24 dist-test-slave-c7tk krb5kdc[6431](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284104, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 21:08:24.941315 7584 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260924 21:08:24.941641 7584 webserver.cc:460] Webserver started at http://127.0.98.3:38997/ using document root <none> and password file <none>
I20260924 21:08:24.941880 7584 fs_manager.cc:359] Metadata directory not provided
I20260924 21:08:24.941951 7584 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 21:08:24.943094 7584 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.001s sys 0.000s
I20260924 21:08:24.943504 7599 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 21:08:24.943635 7584 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 21:08:24.943712 7584 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790284076800039-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790284076800039-392-0/minicluster-data/ts-2/wal
uuid: "54766fc07b09440682225ff08c6814db"
format_stamp: "Formatted at 2026-09-24 21:08:23 on dist-test-slave-c7tk"
server_key: "d52417912803c1748e764c24c07463ab"
server_key_iv: "d05996b569d88ed04b52c0e475301a0a"
server_key_version: "encryptionkey@0"
I20260924 21:08:24.943965 7584 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790284076800039-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790284076800039-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790284076800039-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 21:08:24.958829 7584 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 21:08:24.960527 7584 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 21:08:24.960666 7584 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 21:08:24.960843 7584 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 21:08:24.961143 7584 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 21:08:24.961179 7584 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 21:08:24.961212 7584 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 21:08:24.961234 7584 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 21:08:24.966180 7584 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:37191
I20260924 21:08:24.966220 7712 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:37191 every 8 connection(s)
I20260924 21:08:24.966496 7584 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790284076800039-392-0/minicluster-data/ts-2/data/info.pb
Sep 24 21:08:24 dist-test-slave-c7tk krb5kdc[6431](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284104, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260924 21:08:24.972893 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAzNNjS/build/release/bin/kudu as pid 7584
Sep 24 21:08:24 dist-test-slave-c7tk krb5kdc[6431](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284104, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
Sep 24 21:08:24 dist-test-slave-c7tk krb5kdc[6431](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284104, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260924 21:08:24.976388 7125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:24.967238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:46007 (local address 127.0.98.61:34067)
0924 21:08:24.967326 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0924 21:08:24.967327 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 21:08:24.967507 (+ 180us) server_negotiation.cc:407] Connection header received
0924 21:08:24.968226 (+ 719us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:08:24.968227 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:08:24.968236 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:08:24.968279 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:08:24.969071 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:24.969828 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:24.971198 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:24.971406 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:24.971467 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:24.973145 (+ 1678us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 21:08:24.973147 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 21:08:24.973147 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 21:08:24.973153 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 21:08:24.974954 (+ 1801us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:08:24.975408 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:08:24.975410 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:08:24.975410 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:08:24.975463 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:08:24.975734 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:08:24.975736 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:08:24.975736 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:08:24.975888 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 21:08:24.975944 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0924 21:08:24.976243 (+ 299us) server_negotiation.cc:299] Negotiation successful
0924 21:08:24.976312 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260924 21:08:24.976804 7719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:24.967034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:34067 (local address 127.0.98.3:46007)
0924 21:08:24.967444 (+ 410us) negotiation.cc:107] Waiting for socket to connect
0924 21:08:24.967466 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0924 21:08:24.968151 (+ 685us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 21:08:24.968330 (+ 179us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 21:08:24.968333 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 21:08:24.968804 (+ 471us) client_negotiation.cc:189] Negotiated authn=SASL
0924 21:08:24.968985 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:24.968990 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:24.969927 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:24.969928 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:24.970806 (+ 878us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:24.970807 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:24.971487 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:24.971488 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:24.971539 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:24.971804 (+ 265us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 21:08:24.971807 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0924 21:08:24.973037 (+ 1230us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 21:08:24.975043 (+ 2006us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 21:08:24.975049 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 21:08:24.975051 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 21:08:24.975319 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 21:08:24.975533 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 21:08:24.975534 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 21:08:24.975535 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 21:08:24.975650 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 21:08:24.975984 (+ 334us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 21:08:24.975989 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 21:08:24.976118 (+ 129us) client_negotiation.cc:769] Sending connection context
0924 21:08:24.976459 (+ 341us) client_negotiation.cc:240] Negotiation successful
0924 21:08:24.976624 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":379,"mutex_wait_us":75,"spinlock_wait_cycles":7040,"thread_start_us":100,"threads_started":1}
I20260924 21:08:24.976974 7311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:24.967082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:57209 (local address 127.0.98.60:39715)
0924 21:08:24.967283 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0924 21:08:24.967285 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 21:08:24.967295 (+ 10us) server_negotiation.cc:407] Connection header received
0924 21:08:24.968254 (+ 959us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:08:24.968255 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:08:24.968266 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:08:24.968293 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:08:24.969089 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:24.969622 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:24.970873 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:24.971117 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:24.971160 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:24.973879 (+ 2719us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 21:08:24.973880 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 21:08:24.973880 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 21:08:24.973887 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 21:08:24.975576 (+ 1689us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:08:24.975968 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:08:24.975969 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:08:24.975969 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:08:24.976017 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:08:24.976169 (+ 152us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:08:24.976169 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:08:24.976170 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:08:24.976271 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 21:08:24.976319 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0924 21:08:24.976857 (+ 538us) server_negotiation.cc:299] Negotiation successful
0924 21:08:24.976926 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":179}
I20260924 21:08:24.977311 7718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:24.967034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39715 (local address 127.0.98.3:57209)
0924 21:08:24.967186 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0924 21:08:24.967220 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0924 21:08:24.968149 (+ 929us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 21:08:24.968352 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 21:08:24.968352 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 21:08:24.968775 (+ 423us) client_negotiation.cc:189] Negotiated authn=SASL
0924 21:08:24.968987 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:24.968993 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:24.969718 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:24.969719 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:24.970802 (+ 1083us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:24.970804 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:24.971174 (+ 370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:24.971174 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:24.971219 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:24.971678 (+ 459us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 21:08:24.971682 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 21:08:24.973780 (+ 2098us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 21:08:24.975647 (+ 1867us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 21:08:24.975648 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 21:08:24.975648 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 21:08:24.975892 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 21:08:24.976071 (+ 179us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 21:08:24.976071 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 21:08:24.976071 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 21:08:24.976115 (+ 44us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 21:08:24.976340 (+ 225us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 21:08:24.976341 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 21:08:24.976381 (+ 40us) client_negotiation.cc:769] Sending connection context
0924 21:08:24.976459 (+ 78us) client_negotiation.cc:240] Negotiation successful
0924 21:08:24.976778 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"mutex_wait_us":269,"thread_start_us":86,"threads_started":1}
I20260924 21:08:24.977595 7715 heartbeater.cc:347] Connected to a master server at 127.0.98.61:34067
I20260924 21:08:24.977598 7713 heartbeater.cc:347] Connected to a master server at 127.0.98.60:39715
I20260924 21:08:24.977684 7715 heartbeater.cc:464] Registering TS with master...
I20260924 21:08:24.977685 7713 heartbeater.cc:464] Registering TS with master...
I20260924 21:08:24.977857 7715 heartbeater.cc:511] Master 127.0.98.61:34067 requested a full tablet report, sending...
I20260924 21:08:24.977898 7713 heartbeater.cc:511] Master 127.0.98.60:39715 requested a full tablet report, sending...
I20260924 21:08:24.978197 7260 ts_manager.cc:194] Registered new tserver with Master: 54766fc07b09440682225ff08c6814db (127.0.98.3:37191)
I20260924 21:08:24.978233 7074 ts_manager.cc:194] Registered new tserver with Master: 54766fc07b09440682225ff08c6814db (127.0.98.3:37191)
I20260924 21:08:24.979703 7717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:24.967034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44697 (local address 127.0.98.3:49243)
0924 21:08:24.967469 (+ 435us) negotiation.cc:107] Waiting for socket to connect
0924 21:08:24.967476 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0924 21:08:24.968163 (+ 687us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 21:08:24.968353 (+ 190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 21:08:24.968353 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 21:08:24.968793 (+ 440us) client_negotiation.cc:189] Negotiated authn=SASL
0924 21:08:24.968987 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:24.968993 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:24.969966 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:24.969967 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:24.970663 (+ 696us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:24.970665 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:24.971057 (+ 392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:24.971058 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:24.971135 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:24.971649 (+ 514us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 21:08:24.971658 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 21:08:24.973291 (+ 1633us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 21:08:24.975618 (+ 2327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 21:08:24.975619 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 21:08:24.975619 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 21:08:24.977705 (+ 2086us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 21:08:24.977995 (+ 290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 21:08:24.977996 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 21:08:24.977996 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 21:08:24.978360 (+ 364us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 21:08:24.978791 (+ 431us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 21:08:24.978792 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 21:08:24.978842 (+ 50us) client_negotiation.cc:769] Sending connection context
0924 21:08:24.979312 (+ 470us) client_negotiation.cc:240] Negotiation successful
0924 21:08:24.979413 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":416,"mutex_wait_us":27,"thread_start_us":86,"threads_started":1}
I20260924 21:08:24.979974 6516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:24.967254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:49243 (local address 127.0.98.62:44697)
0924 21:08:24.967303 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0924 21:08:24.967305 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 21:08:24.967519 (+ 214us) server_negotiation.cc:407] Connection header received
0924 21:08:24.968254 (+ 735us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:08:24.968255 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:08:24.968266 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:08:24.968293 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:08:24.969110 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:24.969880 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:24.970742 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:24.970992 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:24.971050 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:24.973389 (+ 2339us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 21:08:24.973390 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 21:08:24.973390 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 21:08:24.973397 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 21:08:24.975529 (+ 2132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:08:24.977787 (+ 2258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:08:24.977788 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:08:24.977788 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:08:24.977866 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:08:24.978554 (+ 688us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:08:24.978555 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:08:24.978556 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:08:24.978706 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 21:08:24.978755 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0924 21:08:24.979813 (+ 1058us) server_negotiation.cc:299] Negotiation successful
0924 21:08:24.979900 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260924 21:08:24.980552 7714 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44697
I20260924 21:08:24.980638 7714 heartbeater.cc:464] Registering TS with master...
I20260924 21:08:24.980726 7714 heartbeater.cc:511] Master 127.0.98.62:44697 requested a full tablet report, sending...
I20260924 21:08:24.980991 6476 ts_manager.cc:194] Re-registered known tserver with Master: 54766fc07b09440682225ff08c6814db (127.0.98.3:37191)
I20260924 21:08:24.981442 6476 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:49243
I20260924 21:08:24.983862 7728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:24.976057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48388 (local address 127.0.98.61:34067)
0924 21:08:24.976227 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0924 21:08:24.976228 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 21:08:24.976343 (+ 115us) server_negotiation.cc:407] Connection header received
0924 21:08:24.976361 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:08:24.976362 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:08:24.976371 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:08:24.976418 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:08:24.977399 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:24.978037 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:24.978981 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:24.979564 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:24.979625 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:24.980924 (+ 1299us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 21:08:24.980925 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 21:08:24.980926 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 21:08:24.980932 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 21:08:24.982361 (+ 1429us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:08:24.982994 (+ 633us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:08:24.982995 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:08:24.982995 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:08:24.983048 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:08:24.983251 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:08:24.983251 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:08:24.983252 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:08:24.983371 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 21:08:24.983467 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0924 21:08:24.983700 (+ 233us) server_negotiation.cc:299] Negotiation successful
0924 21:08:24.983774 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":156,"thread_start_us":37,"threads_started":1}
I20260924 21:08:24.984397 7724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:24.974023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35200 (local address 127.0.98.60:39715)
0924 21:08:24.974102 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0924 21:08:24.974103 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 21:08:24.974728 (+ 625us) server_negotiation.cc:407] Connection header received
0924 21:08:24.974747 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:08:24.974748 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:08:24.974758 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:08:24.974808 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:08:24.975666 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:24.976322 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:24.978688 (+ 2366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:24.978990 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:24.979041 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:24.980730 (+ 1689us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 21:08:24.980731 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 21:08:24.980732 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 21:08:24.980738 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 21:08:24.983188 (+ 2450us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:08:24.983544 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:08:24.983545 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:08:24.983546 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:08:24.983598 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:08:24.983786 (+ 188us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:08:24.983787 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:08:24.983787 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:08:24.983904 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 21:08:24.984062 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0924 21:08:24.984219 (+ 157us) server_negotiation.cc:299] Negotiation successful
0924 21:08:24.984294 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64,"thread_start_us":47,"threads_started":1}
I20260924 21:08:24.985831 7726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:24.974264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52330 (local address 127.0.98.62:44697)
0924 21:08:24.978604 (+ 4340us) server_negotiation.cc:206] Beginning negotiation
0924 21:08:24.978606 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 21:08:24.978611 (+ 5us) server_negotiation.cc:407] Connection header received
0924 21:08:24.978627 (+ 16us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:08:24.978628 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:08:24.978639 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:08:24.979855 (+ 1216us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:08:24.979883 (+ 28us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:24.980522 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:24.981470 (+ 948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:24.981759 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:24.981883 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:24.983281 (+ 1398us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 21:08:24.983282 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 21:08:24.983282 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 21:08:24.983288 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 21:08:24.984921 (+ 1633us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:08:24.985192 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:08:24.985192 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:08:24.985193 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:08:24.985238 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:08:24.985406 (+ 168us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:08:24.985406 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:08:24.985407 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:08:24.985500 (+ 93us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 21:08:24.985585 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0924 21:08:24.985663 (+ 78us) server_negotiation.cc:299] Negotiation successful
0924 21:08:24.985728 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4313,"thread_start_us":104,"threads_started":1}
I20260924 21:08:24.987110 6476 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52330:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 21:08:24.988484 6476 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 21:08:24.995213 7034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:24.991846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:37191 (local address 127.0.0.1:42354)
0924 21:08:24.992052 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0924 21:08:24.992061 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 21:08:24.992091 (+ 30us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 21:08:24.992423 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 21:08:24.992424 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 21:08:24.992426 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 21:08:24.992461 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:24.992462 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:24.993515 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:24.993516 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:24.994417 (+ 901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:24.994420 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:24.995017 (+ 597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:24.995018 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:24.995070 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:24.995073 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 21:08:24.995124 (+ 51us) client_negotiation.cc:240] Negotiation successful
0924 21:08:24.995160 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":192,"mutex_wait_us":2}
I20260924 21:08:24.995220 7734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:24.991935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42354 (local address 127.0.98.3:37191)
0924 21:08:24.992223 (+ 288us) server_negotiation.cc:206] Beginning negotiation
0924 21:08:24.992226 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 21:08:24.992236 (+ 10us) server_negotiation.cc:407] Connection header received
0924 21:08:24.992268 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:08:24.992269 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:08:24.992286 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:08:24.992422 (+ 136us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 21:08:24.992555 (+ 133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:24.993436 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:24.994498 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:24.994941 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:24.995018 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:24.995077 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0924 21:08:24.995134 (+ 57us) server_negotiation.cc:299] Negotiation successful
0924 21:08:24.995160 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":262,"thread_start_us":69,"threads_started":1}
I20260924 21:08:24.995721 7647 tablet_service.cc:1467] Processing CreateTablet for tablet 74130c94d96b4676b69e5942806ebad5 (DEFAULT_TABLE table=test-table [id=0f5267ecdfa2413cb6f0e196df9898bd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 21:08:24.996002 7647 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 74130c94d96b4676b69e5942806ebad5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 21:08:24.996088 7735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:24.991960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41965 (local address 127.0.0.1:46502)
0924 21:08:24.992478 (+ 518us) negotiation.cc:107] Waiting for socket to connect
0924 21:08:24.992487 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 21:08:24.992533 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 21:08:24.992911 (+ 378us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 21:08:24.992911 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 21:08:24.992913 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 21:08:24.992943 (+ 30us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:24.992944 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:24.993772 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:24.993773 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:24.994969 (+ 1196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:24.994971 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:24.995859 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:24.995860 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:24.995918 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:24.995922 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 21:08:24.995966 (+ 44us) client_negotiation.cc:240] Negotiation successful
0924 21:08:24.996000 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":503,"mutex_wait_us":28,"thread_start_us":61,"threads_started":1}
I20260924 21:08:24.996181 7738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:24.992236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46502 (local address 127.0.98.1:41965)
0924 21:08:24.992776 (+ 540us) server_negotiation.cc:206] Beginning negotiation
0924 21:08:24.992780 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 21:08:24.992787 (+ 7us) server_negotiation.cc:407] Connection header received
0924 21:08:24.992818 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:08:24.992820 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:08:24.992834 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:08:24.992912 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 21:08:24.992991 (+ 79us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:24.993689 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:24.995057 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:24.995782 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:24.995888 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:24.995951 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0924 21:08:24.995975 (+ 24us) server_negotiation.cc:299] Negotiation successful
0924 21:08:24.996018 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":508,"thread_start_us":103,"threads_started":1}
I20260924 21:08:24.996224 7736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:24.991940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:46665 (local address 127.0.0.1:34376)
0924 21:08:24.992267 (+ 327us) negotiation.cc:107] Waiting for socket to connect
0924 21:08:24.992276 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 21:08:24.992334 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 21:08:24.992674 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 21:08:24.992675 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 21:08:24.992676 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 21:08:24.992711 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:24.992712 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:24.993800 (+ 1088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:24.993800 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:24.994969 (+ 1169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:24.994971 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:24.995909 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:24.995910 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:24.995966 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:24.995969 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 21:08:24.996007 (+ 38us) client_negotiation.cc:240] Negotiation successful
0924 21:08:24.996040 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":314,"mutex_wait_us":54,"thread_start_us":68,"threads_started":1}
I20260924 21:08:24.996649 7737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:24.992016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34376 (local address 127.0.98.2:46665)
0924 21:08:24.992541 (+ 525us) server_negotiation.cc:206] Beginning negotiation
0924 21:08:24.992545 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 21:08:24.992554 (+ 9us) server_negotiation.cc:407] Connection header received
0924 21:08:24.992589 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:08:24.992591 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:08:24.992604 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:08:24.992667 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 21:08:24.992786 (+ 119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:24.993707 (+ 921us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:24.995056 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:24.995780 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:24.995843 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:24.996527 (+ 684us) server_negotiation.cc:1035] Waiting for connection context
0924 21:08:24.996549 (+ 22us) server_negotiation.cc:299] Negotiation successful
0924 21:08:24.996583 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":477,"thread_start_us":68,"threads_started":1}
I20260924 21:08:24.996680 7375 tablet_service.cc:1467] Processing CreateTablet for tablet 74130c94d96b4676b69e5942806ebad5 (DEFAULT_TABLE table=test-table [id=0f5267ecdfa2413cb6f0e196df9898bd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 21:08:24.996968 7375 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 74130c94d96b4676b69e5942806ebad5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 21:08:24.997112 7511 tablet_service.cc:1467] Processing CreateTablet for tablet 74130c94d96b4676b69e5942806ebad5 (DEFAULT_TABLE table=test-table [id=0f5267ecdfa2413cb6f0e196df9898bd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 21:08:24.997390 7511 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 74130c94d96b4676b69e5942806ebad5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 21:08:24.998762 7739 tablet_bootstrap.cc:492] T 74130c94d96b4676b69e5942806ebad5 P 54766fc07b09440682225ff08c6814db: Bootstrap starting.
I20260924 21:08:24.999402 7739 tablet_bootstrap.cc:654] T 74130c94d96b4676b69e5942806ebad5 P 54766fc07b09440682225ff08c6814db: Neither blocks nor log segments found. Creating new log.
I20260924 21:08:24.999680 7739 log.cc:824] T 74130c94d96b4676b69e5942806ebad5 P 54766fc07b09440682225ff08c6814db: Log is configured to *not* fsync() on all Append() calls
I20260924 21:08:25.000360 7739 tablet_bootstrap.cc:492] T 74130c94d96b4676b69e5942806ebad5 P 54766fc07b09440682225ff08c6814db: No bootstrap required, opened a new log
I20260924 21:08:25.000424 7739 ts_tablet_manager.cc:1397] T 74130c94d96b4676b69e5942806ebad5 P 54766fc07b09440682225ff08c6814db: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 21:08:25.000747 7742 tablet_bootstrap.cc:492] T 74130c94d96b4676b69e5942806ebad5 P 80a0113d8614402db4980fda67be9b4f: Bootstrap starting.
I20260924 21:08:25.001287 7742 tablet_bootstrap.cc:654] T 74130c94d96b4676b69e5942806ebad5 P 80a0113d8614402db4980fda67be9b4f: Neither blocks nor log segments found. Creating new log.
I20260924 21:08:25.001927 7741 tablet_bootstrap.cc:492] T 74130c94d96b4676b69e5942806ebad5 P 1111a5950630489ba30bd8b925f7a15b: Bootstrap starting.
I20260924 21:08:25.002182 7742 log.cc:824] T 74130c94d96b4676b69e5942806ebad5 P 80a0113d8614402db4980fda67be9b4f: Log is configured to *not* fsync() on all Append() calls
I20260924 21:08:25.002444 7741 tablet_bootstrap.cc:654] T 74130c94d96b4676b69e5942806ebad5 P 1111a5950630489ba30bd8b925f7a15b: Neither blocks nor log segments found. Creating new log.
I20260924 21:08:25.002712 7741 log.cc:824] T 74130c94d96b4676b69e5942806ebad5 P 1111a5950630489ba30bd8b925f7a15b: Log is configured to *not* fsync() on all Append() calls
I20260924 21:08:25.002895 7742 tablet_bootstrap.cc:492] T 74130c94d96b4676b69e5942806ebad5 P 80a0113d8614402db4980fda67be9b4f: No bootstrap required, opened a new log
I20260924 21:08:25.002957 7742 ts_tablet_manager.cc:1397] T 74130c94d96b4676b69e5942806ebad5 P 80a0113d8614402db4980fda67be9b4f: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260924 21:08:25.003021 7739 raft_consensus.cc:348] T 74130c94d96b4676b69e5942806ebad5 P 54766fc07b09440682225ff08c6814db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54766fc07b09440682225ff08c6814db" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37191 } } peers { permanent_uuid: "80a0113d8614402db4980fda67be9b4f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46665 } } peers { permanent_uuid: "1111a5950630489ba30bd8b925f7a15b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41965 } }
I20260924 21:08:25.003150 7739 raft_consensus.cc:374] T 74130c94d96b4676b69e5942806ebad5 P 54766fc07b09440682225ff08c6814db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 21:08:25.003181 7739 raft_consensus.cc:729] T 74130c94d96b4676b69e5942806ebad5 P 54766fc07b09440682225ff08c6814db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54766fc07b09440682225ff08c6814db, State: Initialized, Role: FOLLOWER
I20260924 21:08:25.003289 7739 consensus_queue.cc:260] T 74130c94d96b4676b69e5942806ebad5 P 54766fc07b09440682225ff08c6814db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54766fc07b09440682225ff08c6814db" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37191 } } peers { permanent_uuid: "80a0113d8614402db4980fda67be9b4f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46665 } } peers { permanent_uuid: "1111a5950630489ba30bd8b925f7a15b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41965 } }
I20260924 21:08:25.003381 7741 tablet_bootstrap.cc:492] T 74130c94d96b4676b69e5942806ebad5 P 1111a5950630489ba30bd8b925f7a15b: No bootstrap required, opened a new log
I20260924 21:08:25.003432 7741 ts_tablet_manager.cc:1397] T 74130c94d96b4676b69e5942806ebad5 P 1111a5950630489ba30bd8b925f7a15b: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 21:08:25.003491 7739 ts_tablet_manager.cc:1428] T 74130c94d96b4676b69e5942806ebad5 P 54766fc07b09440682225ff08c6814db: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 21:08:25.003593 7714 heartbeater.cc:503] Master 127.0.98.62:44697 was elected leader, sending a full tablet report...
I20260924 21:08:25.005155 7742 raft_consensus.cc:348] T 74130c94d96b4676b69e5942806ebad5 P 80a0113d8614402db4980fda67be9b4f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54766fc07b09440682225ff08c6814db" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37191 } } peers { permanent_uuid: "80a0113d8614402db4980fda67be9b4f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46665 } } peers { permanent_uuid: "1111a5950630489ba30bd8b925f7a15b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41965 } }
I20260924 21:08:25.005272 7742 raft_consensus.cc:374] T 74130c94d96b4676b69e5942806ebad5 P 80a0113d8614402db4980fda67be9b4f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 21:08:25.005309 7742 raft_consensus.cc:729] T 74130c94d96b4676b69e5942806ebad5 P 80a0113d8614402db4980fda67be9b4f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80a0113d8614402db4980fda67be9b4f, State: Initialized, Role: FOLLOWER
I20260924 21:08:25.005255 7741 raft_consensus.cc:348] T 74130c94d96b4676b69e5942806ebad5 P 1111a5950630489ba30bd8b925f7a15b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54766fc07b09440682225ff08c6814db" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37191 } } peers { permanent_uuid: "80a0113d8614402db4980fda67be9b4f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46665 } } peers { permanent_uuid: "1111a5950630489ba30bd8b925f7a15b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41965 } }
I20260924 21:08:25.005398 7741 raft_consensus.cc:374] T 74130c94d96b4676b69e5942806ebad5 P 1111a5950630489ba30bd8b925f7a15b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 21:08:25.005441 7741 raft_consensus.cc:729] T 74130c94d96b4676b69e5942806ebad5 P 1111a5950630489ba30bd8b925f7a15b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1111a5950630489ba30bd8b925f7a15b, State: Initialized, Role: FOLLOWER
I20260924 21:08:25.005447 7742 consensus_queue.cc:260] T 74130c94d96b4676b69e5942806ebad5 P 80a0113d8614402db4980fda67be9b4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54766fc07b09440682225ff08c6814db" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37191 } } peers { permanent_uuid: "80a0113d8614402db4980fda67be9b4f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46665 } } peers { permanent_uuid: "1111a5950630489ba30bd8b925f7a15b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41965 } }
I20260924 21:08:25.005568 7741 consensus_queue.cc:260] T 74130c94d96b4676b69e5942806ebad5 P 1111a5950630489ba30bd8b925f7a15b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54766fc07b09440682225ff08c6814db" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37191 } } peers { permanent_uuid: "80a0113d8614402db4980fda67be9b4f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46665 } } peers { permanent_uuid: "1111a5950630489ba30bd8b925f7a15b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41965 } }
I20260924 21:08:25.005672 7742 ts_tablet_manager.cc:1428] T 74130c94d96b4676b69e5942806ebad5 P 80a0113d8614402db4980fda67be9b4f: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260924 21:08:25.005748 7578 heartbeater.cc:503] Master 127.0.98.62:44697 was elected leader, sending a full tablet report...
I20260924 21:08:25.005823 7741 ts_tablet_manager.cc:1428] T 74130c94d96b4676b69e5942806ebad5 P 1111a5950630489ba30bd8b925f7a15b: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260924 21:08:25.006403 7442 heartbeater.cc:503] Master 127.0.98.62:44697 was elected leader, sending a full tablet report...
I20260924 21:08:25.043705 7736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:25.041534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39715 (local address 127.0.0.1:35204)
0924 21:08:25.041582 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0924 21:08:25.041593 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 21:08:25.041647 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 21:08:25.041820 (+ 173us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 21:08:25.041821 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 21:08:25.041822 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 21:08:25.041861 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:25.041862 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:25.042419 (+ 557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:25.042420 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:25.043021 (+ 601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:25.043023 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:25.043504 (+ 481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:25.043505 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:25.043555 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:25.043558 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 21:08:25.043608 (+ 50us) client_negotiation.cc:240] Negotiation successful
0924 21:08:25.043646 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260924 21:08:25.043701 7724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:25.041616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35204 (local address 127.0.98.60:39715)
0924 21:08:25.041669 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0924 21:08:25.041670 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 21:08:25.041687 (+ 17us) server_negotiation.cc:407] Connection header received
0924 21:08:25.041712 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:08:25.041713 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:08:25.041727 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:08:25.041782 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 21:08:25.041919 (+ 137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:25.042334 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:25.043096 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:25.043423 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:25.043503 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:25.043559 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0924 21:08:25.043614 (+ 55us) server_negotiation.cc:299] Negotiation successful
0924 21:08:25.043647 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260924 21:08:25.044027 7270 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 65393c0abc864ec4b9d3d76877a9a863 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.9] Dedup: 1.7->[1.8-1.9]
I20260924 21:08:25.044746 7299 sys_catalog.cc:455] T 00000000000000000000000000000000 P 65393c0abc864ec4b9d3d76877a9a863 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0f53f4b63d8e4da0905f8db79d1fc7db. Latest consensus state: current_term: 1 leader_uuid: "0f53f4b63d8e4da0905f8db79d1fc7db" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "0f53f4b63d8e4da0905f8db79d1fc7db" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44697 } } peers { permanent_uuid: "a5110fb5b32e4973948078ef72376369" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34067 } attrs { promote: false } } peers { permanent_uuid: "65393c0abc864ec4b9d3d76877a9a863" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 39715 } attrs { promote: true } } }
I20260924 21:08:25.044833 7299 sys_catalog.cc:458] T 00000000000000000000000000000000 P 65393c0abc864ec4b9d3d76877a9a863 [sys.catalog]: This master's current role is: LEARNER
W20260924 21:08:25.100683 7580 tablet.cc:2369] T 74130c94d96b4676b69e5942806ebad5 P 80a0113d8614402db4980fda67be9b4f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 21:08:25.136938 7747 raft_consensus.cc:482] T 74130c94d96b4676b69e5942806ebad5 P 1111a5950630489ba30bd8b925f7a15b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 21:08:25.137039 7747 raft_consensus.cc:504] T 74130c94d96b4676b69e5942806ebad5 P 1111a5950630489ba30bd8b925f7a15b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54766fc07b09440682225ff08c6814db" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37191 } } peers { permanent_uuid: "80a0113d8614402db4980fda67be9b4f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46665 } } peers { permanent_uuid: "1111a5950630489ba30bd8b925f7a15b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41965 } }
I20260924 21:08:25.137398 7747 leader_election.cc:288] T 74130c94d96b4676b69e5942806ebad5 P 1111a5950630489ba30bd8b925f7a15b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 54766fc07b09440682225ff08c6814db (127.0.98.3:37191), 80a0113d8614402db4980fda67be9b4f (127.0.98.2:46665)
I20260924 21:08:25.140152 7734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:25.137578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:35847 (local address 127.0.98.3:37191)
0924 21:08:25.137626 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0924 21:08:25.137628 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 21:08:25.137635 (+ 7us) server_negotiation.cc:407] Connection header received
0924 21:08:25.137659 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:08:25.137660 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:08:25.137670 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:08:25.137741 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 21:08:25.137929 (+ 188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:25.138365 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:25.139380 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:25.139842 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:25.139929 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:25.139938 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0924 21:08:25.140064 (+ 126us) server_negotiation.cc:299] Negotiation successful
0924 21:08:25.140093 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260924 21:08:25.140182 7446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:25.137507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:37191 (local address 127.0.98.1:35847)
0924 21:08:25.137551 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0924 21:08:25.137562 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 21:08:25.137605 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 21:08:25.137744 (+ 139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 21:08:25.137744 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 21:08:25.137745 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 21:08:25.137834 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:25.137836 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:25.138443 (+ 607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:25.138444 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:25.139279 (+ 835us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:25.139281 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:25.139928 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:25.139929 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:25.139993 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:25.139995 (+ 2us) client_negotiation.cc:769] Sending connection context
0924 21:08:25.140057 (+ 62us) client_negotiation.cc:240] Negotiation successful
0924 21:08:25.140092 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25,"mutex_wait_us":20}
I20260924 21:08:25.140198 7737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:25.137624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:55415 (local address 127.0.98.2:46665)
0924 21:08:25.137673 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0924 21:08:25.137674 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 21:08:25.137681 (+ 7us) server_negotiation.cc:407] Connection header received
0924 21:08:25.137704 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:08:25.137705 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:08:25.137714 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:08:25.137769 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 21:08:25.137954 (+ 185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:25.138403 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:25.139441 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:25.139866 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:25.139952 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:25.139960 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0924 21:08:25.140101 (+ 141us) server_negotiation.cc:299] Negotiation successful
0924 21:08:25.140131 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260924 21:08:25.140194 7445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:25.137544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:46665 (local address 127.0.98.1:55415)
0924 21:08:25.137588 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0924 21:08:25.137598 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 21:08:25.137632 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 21:08:25.137780 (+ 148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 21:08:25.137781 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 21:08:25.137782 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 21:08:25.137887 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:25.137889 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:25.138485 (+ 596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:25.138487 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:25.139352 (+ 865us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:25.139355 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:25.139954 (+ 599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:25.139955 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:25.140017 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:25.140019 (+ 2us) client_negotiation.cc:769] Sending connection context
0924 21:08:25.140081 (+ 62us) client_negotiation.cc:240] Negotiation successful
0924 21:08:25.140115 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28}
I20260924 21:08:25.140492 7667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74130c94d96b4676b69e5942806ebad5" candidate_uuid: "1111a5950630489ba30bd8b925f7a15b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54766fc07b09440682225ff08c6814db" is_pre_election: true
I20260924 21:08:25.140489 7531 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74130c94d96b4676b69e5942806ebad5" candidate_uuid: "1111a5950630489ba30bd8b925f7a15b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80a0113d8614402db4980fda67be9b4f" is_pre_election: true
I20260924 21:08:25.140625 7667 raft_consensus.cc:2454] T 74130c94d96b4676b69e5942806ebad5 P 54766fc07b09440682225ff08c6814db [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1111a5950630489ba30bd8b925f7a15b in term 0.
I20260924 21:08:25.140614 7531 raft_consensus.cc:2454] T 74130c94d96b4676b69e5942806ebad5 P 80a0113d8614402db4980fda67be9b4f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1111a5950630489ba30bd8b925f7a15b in term 0.
I20260924 21:08:25.140834 7328 leader_election.cc:302] T 74130c94d96b4676b69e5942806ebad5 P 1111a5950630489ba30bd8b925f7a15b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1111a5950630489ba30bd8b925f7a15b, 54766fc07b09440682225ff08c6814db; no voters:
I20260924 21:08:25.140957 7747 raft_consensus.cc:2781] T 74130c94d96b4676b69e5942806ebad5 P 1111a5950630489ba30bd8b925f7a15b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 21:08:25.141003 7747 raft_consensus.cc:482] T 74130c94d96b4676b69e5942806ebad5 P 1111a5950630489ba30bd8b925f7a15b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 21:08:25.141022 7747 raft_consensus.cc:3037] T 74130c94d96b4676b69e5942806ebad5 P 1111a5950630489ba30bd8b925f7a15b [term 0 FOLLOWER]: Advancing to term 1
I20260924 21:08:25.141562 7747 raft_consensus.cc:504] T 74130c94d96b4676b69e5942806ebad5 P 1111a5950630489ba30bd8b925f7a15b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54766fc07b09440682225ff08c6814db" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37191 } } peers { permanent_uuid: "80a0113d8614402db4980fda67be9b4f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46665 } } peers { permanent_uuid: "1111a5950630489ba30bd8b925f7a15b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41965 } }
I20260924 21:08:25.141673 7747 leader_election.cc:288] T 74130c94d96b4676b69e5942806ebad5 P 1111a5950630489ba30bd8b925f7a15b [CANDIDATE]: Term 1 election: Requested vote from peers 54766fc07b09440682225ff08c6814db (127.0.98.3:37191), 80a0113d8614402db4980fda67be9b4f (127.0.98.2:46665)
I20260924 21:08:25.141824 7531 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74130c94d96b4676b69e5942806ebad5" candidate_uuid: "1111a5950630489ba30bd8b925f7a15b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80a0113d8614402db4980fda67be9b4f"
I20260924 21:08:25.141835 7667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74130c94d96b4676b69e5942806ebad5" candidate_uuid: "1111a5950630489ba30bd8b925f7a15b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54766fc07b09440682225ff08c6814db"
I20260924 21:08:25.141892 7531 raft_consensus.cc:3037] T 74130c94d96b4676b69e5942806ebad5 P 80a0113d8614402db4980fda67be9b4f [term 0 FOLLOWER]: Advancing to term 1
I20260924 21:08:25.141898 7667 raft_consensus.cc:3037] T 74130c94d96b4676b69e5942806ebad5 P 54766fc07b09440682225ff08c6814db [term 0 FOLLOWER]: Advancing to term 1
I20260924 21:08:25.142644 7667 raft_consensus.cc:2454] T 74130c94d96b4676b69e5942806ebad5 P 54766fc07b09440682225ff08c6814db [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1111a5950630489ba30bd8b925f7a15b in term 1.
I20260924 21:08:25.142644 7531 raft_consensus.cc:2454] T 74130c94d96b4676b69e5942806ebad5 P 80a0113d8614402db4980fda67be9b4f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1111a5950630489ba30bd8b925f7a15b in term 1.
I20260924 21:08:25.142786 7328 leader_election.cc:302] T 74130c94d96b4676b69e5942806ebad5 P 1111a5950630489ba30bd8b925f7a15b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1111a5950630489ba30bd8b925f7a15b, 54766fc07b09440682225ff08c6814db; no voters:
I20260924 21:08:25.142879 7747 raft_consensus.cc:2781] T 74130c94d96b4676b69e5942806ebad5 P 1111a5950630489ba30bd8b925f7a15b [term 1 FOLLOWER]: Leader election won for term 1
I20260924 21:08:25.142997 7747 raft_consensus.cc:686] T 74130c94d96b4676b69e5942806ebad5 P 1111a5950630489ba30bd8b925f7a15b [term 1 LEADER]: Becoming Leader. State: Replica: 1111a5950630489ba30bd8b925f7a15b, State: Running, Role: LEADER
I20260924 21:08:25.143085 7747 consensus_queue.cc:237] T 74130c94d96b4676b69e5942806ebad5 P 1111a5950630489ba30bd8b925f7a15b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54766fc07b09440682225ff08c6814db" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37191 } } peers { permanent_uuid: "80a0113d8614402db4980fda67be9b4f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46665 } } peers { permanent_uuid: "1111a5950630489ba30bd8b925f7a15b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41965 } }
I20260924 21:08:25.143846 6475 catalog_manager.cc:5495] T 74130c94d96b4676b69e5942806ebad5 P 1111a5950630489ba30bd8b925f7a15b reported cstate change: term changed from 0 to 1, leader changed from <none> to 1111a5950630489ba30bd8b925f7a15b (127.0.98.1). New cstate: current_term: 1 leader_uuid: "1111a5950630489ba30bd8b925f7a15b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54766fc07b09440682225ff08c6814db" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37191 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "80a0113d8614402db4980fda67be9b4f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46665 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1111a5950630489ba30bd8b925f7a15b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41965 } health_report { overall_health: HEALTHY } } }
I20260924 21:08:25.144332 7223 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0f53f4b63d8e4da0905f8db79d1fc7db [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0f53f4b63d8e4da0905f8db79d1fc7db" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "0f53f4b63d8e4da0905f8db79d1fc7db" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44697 } } peers { permanent_uuid: "a5110fb5b32e4973948078ef72376369" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34067 } attrs { promote: false } } peers { permanent_uuid: "65393c0abc864ec4b9d3d76877a9a863" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 39715 } attrs { promote: true } } }
I20260924 21:08:25.144411 7223 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0f53f4b63d8e4da0905f8db79d1fc7db [sys.catalog]: This master's current role is: LEADER
I20260924 21:08:25.144840 7035 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 0f53f4b63d8e4da0905f8db79d1fc7db: attempting to promote NON_VOTER 65393c0abc864ec4b9d3d76877a9a863 to VOTER
I20260924 21:08:25.144968 7035 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0f53f4b63d8e4da0905f8db79d1fc7db [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "0f53f4b63d8e4da0905f8db79d1fc7db" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44697 } } peers { permanent_uuid: "a5110fb5b32e4973948078ef72376369" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34067 } attrs { promote: false } } peers { permanent_uuid: "65393c0abc864ec4b9d3d76877a9a863" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39715 } attrs { promote: false } }
I20260924 21:08:25.145310 7270 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 65393c0abc864ec4b9d3d76877a9a863 [term 1 LEARNER]: Refusing update from remote peer 0f53f4b63d8e4da0905f8db79d1fc7db: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20260924 21:08:25.145310 7084 raft_consensus.cc:1261] T 00000000000000000000000000000000 P a5110fb5b32e4973948078ef72376369 [term 1 FOLLOWER]: Refusing update from remote peer 0f53f4b63d8e4da0905f8db79d1fc7db: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20260924 21:08:25.145483 7755 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 0f53f4b63d8e4da0905f8db79d1fc7db [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5110fb5b32e4973948078ef72376369" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34067 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20260924 21:08:25.145581 7036 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 0f53f4b63d8e4da0905f8db79d1fc7db [LEADER]: Connected to new peer: Peer: permanent_uuid: "65393c0abc864ec4b9d3d76877a9a863" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39715 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.000s
I20260924 21:08:25.146168 7223 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 0f53f4b63d8e4da0905f8db79d1fc7db [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 7 to 11, 65393c0abc864ec4b9d3d76877a9a863 (127.0.98.60) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "0f53f4b63d8e4da0905f8db79d1fc7db" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44697 } } peers { permanent_uuid: "a5110fb5b32e4973948078ef72376369" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34067 } attrs { promote: false } } peers { permanent_uuid: "65393c0abc864ec4b9d3d76877a9a863" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39715 } attrs { promote: false } } }
I20260924 21:08:25.146301 7270 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 65393c0abc864ec4b9d3d76877a9a863 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 7 to 11, 65393c0abc864ec4b9d3d76877a9a863 (127.0.98.60) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "0f53f4b63d8e4da0905f8db79d1fc7db" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44697 } } peers { permanent_uuid: "a5110fb5b32e4973948078ef72376369" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34067 } attrs { promote: false } } peers { permanent_uuid: "65393c0abc864ec4b9d3d76877a9a863" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39715 } attrs { promote: false } } }
I20260924 21:08:25.146903 7223 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0f53f4b63d8e4da0905f8db79d1fc7db [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "0f53f4b63d8e4da0905f8db79d1fc7db" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "0f53f4b63d8e4da0905f8db79d1fc7db" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44697 } } peers { permanent_uuid: "a5110fb5b32e4973948078ef72376369" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34067 } attrs { promote: false } } peers { permanent_uuid: "65393c0abc864ec4b9d3d76877a9a863" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39715 } attrs { promote: false } } }
I20260924 21:08:25.146967 7223 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0f53f4b63d8e4da0905f8db79d1fc7db [sys.catalog]: This master's current role is: LEADER
I20260924 21:08:25.147014 7223 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0f53f4b63d8e4da0905f8db79d1fc7db [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0f53f4b63d8e4da0905f8db79d1fc7db" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "0f53f4b63d8e4da0905f8db79d1fc7db" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44697 } } peers { permanent_uuid: "a5110fb5b32e4973948078ef72376369" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34067 } attrs { promote: false } } peers { permanent_uuid: "65393c0abc864ec4b9d3d76877a9a863" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39715 } attrs { promote: false } } }
I20260924 21:08:25.147054 7223 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0f53f4b63d8e4da0905f8db79d1fc7db [sys.catalog]: This master's current role is: LEADER
I20260924 21:08:25.147056 7755 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0f53f4b63d8e4da0905f8db79d1fc7db [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0f53f4b63d8e4da0905f8db79d1fc7db" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "0f53f4b63d8e4da0905f8db79d1fc7db" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44697 } } peers { permanent_uuid: "a5110fb5b32e4973948078ef72376369" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34067 } attrs { promote: false } } peers { permanent_uuid: "65393c0abc864ec4b9d3d76877a9a863" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39715 } attrs { promote: false } } }
I20260924 21:08:25.147163 7755 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0f53f4b63d8e4da0905f8db79d1fc7db [sys.catalog]: This master's current role is: LEADER
I20260924 21:08:25.147137 7084 raft_consensus.cc:2932] T 00000000000000000000000000000000 P a5110fb5b32e4973948078ef72376369 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 7 to 11, 65393c0abc864ec4b9d3d76877a9a863 (127.0.98.60) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "0f53f4b63d8e4da0905f8db79d1fc7db" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44697 } } peers { permanent_uuid: "a5110fb5b32e4973948078ef72376369" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34067 } attrs { promote: false } } peers { permanent_uuid: "65393c0abc864ec4b9d3d76877a9a863" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39715 } attrs { promote: false } } }
I20260924 21:08:25.147189 7299 sys_catalog.cc:455] T 00000000000000000000000000000000 P 65393c0abc864ec4b9d3d76877a9a863 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0f53f4b63d8e4da0905f8db79d1fc7db" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "0f53f4b63d8e4da0905f8db79d1fc7db" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44697 } } peers { permanent_uuid: "a5110fb5b32e4973948078ef72376369" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34067 } attrs { promote: false } } peers { permanent_uuid: "65393c0abc864ec4b9d3d76877a9a863" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39715 } attrs { promote: false } } }
I20260924 21:08:25.147254 7299 sys_catalog.cc:458] T 00000000000000000000000000000000 P 65393c0abc864ec4b9d3d76877a9a863 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 21:08:25.147886 7756 sys_catalog.cc:455] T 00000000000000000000000000000000 P a5110fb5b32e4973948078ef72376369 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0f53f4b63d8e4da0905f8db79d1fc7db" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "0f53f4b63d8e4da0905f8db79d1fc7db" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44697 } } peers { permanent_uuid: "a5110fb5b32e4973948078ef72376369" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34067 } attrs { promote: false } } peers { permanent_uuid: "65393c0abc864ec4b9d3d76877a9a863" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39715 } attrs { promote: false } } }
I20260924 21:08:25.147966 7756 sys_catalog.cc:458] T 00000000000000000000000000000000 P a5110fb5b32e4973948078ef72376369 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 21:08:25.176626 7738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:25.174551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46516 (local address 127.0.98.1:41965)
0924 21:08:25.174610 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0924 21:08:25.174612 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 21:08:25.174621 (+ 9us) server_negotiation.cc:407] Connection header received
0924 21:08:25.174691 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:08:25.174692 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:08:25.174715 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:08:25.174782 (+ 67us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 21:08:25.174920 (+ 138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:25.175314 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:25.175973 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:25.176143 (+ 170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:25.176210 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:25.176342 (+ 132us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 21:08:25.176441 (+ 99us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 21:08:25.176524 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0924 21:08:25.176540 (+ 16us) server_negotiation.cc:299] Negotiation successful
0924 21:08:25.176555 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260924 21:08:25.178498 7667 raft_consensus.cc:1261] T 74130c94d96b4676b69e5942806ebad5 P 54766fc07b09440682225ff08c6814db [term 1 FOLLOWER]: Refusing update from remote peer 1111a5950630489ba30bd8b925f7a15b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 21:08:25.178524 7531 raft_consensus.cc:1261] T 74130c94d96b4676b69e5942806ebad5 P 80a0113d8614402db4980fda67be9b4f [term 1 FOLLOWER]: Refusing update from remote peer 1111a5950630489ba30bd8b925f7a15b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 21:08:25.178879 7752 consensus_queue.cc:1035] T 74130c94d96b4676b69e5942806ebad5 P 1111a5950630489ba30bd8b925f7a15b [LEADER]: Connected to new peer: Peer: permanent_uuid: "54766fc07b09440682225ff08c6814db" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37191 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 21:08:25.179013 7747 consensus_queue.cc:1035] T 74130c94d96b4676b69e5942806ebad5 P 1111a5950630489ba30bd8b925f7a15b [LEADER]: Connected to new peer: Peer: permanent_uuid: "80a0113d8614402db4980fda67be9b4f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46665 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 21:08:25.181893 7760 mvcc.cc:205] Tried to move back new op lower bound from 7333003694808743936 to 7333003694666874880. Current Snapshot: MvccSnapshot[applied={T|T < 7333003694808743936}]
I20260924 21:08:25.182546 7758 mvcc.cc:205] Tried to move back new op lower bound from 7333003694808743936 to 7333003694666874880. Current Snapshot: MvccSnapshot[applied={T|T < 7333003694808743936}]
I20260924 21:08:25.184363 6475 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52330:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 21:08:25.184445 6475 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52330:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 21:08:25.185469 6475 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0f53f4b63d8e4da0905f8db79d1fc7db: Sending DeleteTablet for 3 replicas of tablet 74130c94d96b4676b69e5942806ebad5
I20260924 21:08:25.185844 7511 tablet_service.cc:1514] Processing DeleteTablet for tablet 74130c94d96b4676b69e5942806ebad5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 21:08:25 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:34376
I20260924 21:08:25.185889 7647 tablet_service.cc:1514] Processing DeleteTablet for tablet 74130c94d96b4676b69e5942806ebad5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 21:08:25 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:42354
I20260924 21:08:25.186041 7375 tablet_service.cc:1514] Processing DeleteTablet for tablet 74130c94d96b4676b69e5942806ebad5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 21:08:25 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:46502
I20260924 21:08:25.186213 7770 tablet_replica.cc:330] T 74130c94d96b4676b69e5942806ebad5 P 1111a5950630489ba30bd8b925f7a15b: stopping tablet replica
I20260924 21:08:25.186347 7770 raft_consensus.cc:2229] T 74130c94d96b4676b69e5942806ebad5 P 1111a5950630489ba30bd8b925f7a15b [term 1 LEADER]: Raft consensus shutting down.
I20260924 21:08:25.186471 7770 raft_consensus.cc:2258] T 74130c94d96b4676b69e5942806ebad5 P 1111a5950630489ba30bd8b925f7a15b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 21:08:25.186568 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAzNNjS/build/release/bin/kudu with pid 6444
I20260924 21:08:25.188764 7770 ts_tablet_manager.cc:1905] T 74130c94d96b4676b69e5942806ebad5 P 1111a5950630489ba30bd8b925f7a15b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 21:08:25.190048 7770 ts_tablet_manager.cc:1918] T 74130c94d96b4676b69e5942806ebad5 P 1111a5950630489ba30bd8b925f7a15b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 21:08:25.190124 7770 log.cc:1196] T 74130c94d96b4676b69e5942806ebad5 P 1111a5950630489ba30bd8b925f7a15b: Deleting WAL directory at /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790284076800039-392-0/minicluster-data/ts-0/wal/wals/74130c94d96b4676b69e5942806ebad5
I20260924 21:08:25.190352 7770 ts_tablet_manager.cc:1939] T 74130c94d96b4676b69e5942806ebad5 P 1111a5950630489ba30bd8b925f7a15b: Deleting consensus metadata
W20260924 21:08:25.193292 7331 connection.cc:535] server connection from 127.0.0.1:46502 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260924 21:08:25.193461 7768 tablet_replica.cc:330] T 74130c94d96b4676b69e5942806ebad5 P 80a0113d8614402db4980fda67be9b4f: stopping tablet replica
I20260924 21:08:25.193578 7769 tablet_replica.cc:330] T 74130c94d96b4676b69e5942806ebad5 P 54766fc07b09440682225ff08c6814db: stopping tablet replica
I20260924 21:08:25.193710 7768 raft_consensus.cc:2229] T 74130c94d96b4676b69e5942806ebad5 P 80a0113d8614402db4980fda67be9b4f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 21:08:25.193825 7768 raft_consensus.cc:2258] T 74130c94d96b4676b69e5942806ebad5 P 80a0113d8614402db4980fda67be9b4f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 21:08:25.193953 7769 raft_consensus.cc:2229] T 74130c94d96b4676b69e5942806ebad5 P 54766fc07b09440682225ff08c6814db [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 21:08:25.194038 7769 raft_consensus.cc:2258] T 74130c94d96b4676b69e5942806ebad5 P 54766fc07b09440682225ff08c6814db [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 21:08:25.194378 7769 ts_tablet_manager.cc:1905] T 74130c94d96b4676b69e5942806ebad5 P 54766fc07b09440682225ff08c6814db: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 21:08:25.194371 7768 ts_tablet_manager.cc:1905] T 74130c94d96b4676b69e5942806ebad5 P 80a0113d8614402db4980fda67be9b4f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 21:08:25.195767 7769 ts_tablet_manager.cc:1918] T 74130c94d96b4676b69e5942806ebad5 P 54766fc07b09440682225ff08c6814db: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 21:08:25.195827 7769 log.cc:1196] T 74130c94d96b4676b69e5942806ebad5 P 54766fc07b09440682225ff08c6814db: Deleting WAL directory at /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790284076800039-392-0/minicluster-data/ts-2/wal/wals/74130c94d96b4676b69e5942806ebad5
I20260924 21:08:25.196041 7769 ts_tablet_manager.cc:1939] T 74130c94d96b4676b69e5942806ebad5 P 54766fc07b09440682225ff08c6814db: Deleting consensus metadata
I20260924 21:08:25.196249 7768 ts_tablet_manager.cc:1918] T 74130c94d96b4676b69e5942806ebad5 P 80a0113d8614402db4980fda67be9b4f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 21:08:25.196376 7768 log.cc:1196] T 74130c94d96b4676b69e5942806ebad5 P 80a0113d8614402db4980fda67be9b4f: Deleting WAL directory at /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790284076800039-392-0/minicluster-data/ts-1/wal/wals/74130c94d96b4676b69e5942806ebad5
W20260924 21:08:25.196605 7603 connection.cc:441] server connection from 127.0.0.1:42354 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42354 (request call id 1) could send its response
I20260924 21:08:25.196759 7768 ts_tablet_manager.cc:1939] T 74130c94d96b4676b69e5942806ebad5 P 80a0113d8614402db4980fda67be9b4f: Deleting consensus metadata
W20260924 21:08:25.197069 7464 connection.cc:441] server connection from 127.0.0.1:34376 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34376 (request call id 1) could send its response
I20260924 21:08:25.202646 7728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:25.194800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48404 (local address 127.0.98.61:34067)
0924 21:08:25.194834 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0924 21:08:25.194836 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 21:08:25.195103 (+ 267us) server_negotiation.cc:407] Connection header received
0924 21:08:25.195183 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:08:25.195183 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:08:25.195193 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:08:25.195217 (+ 24us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:08:25.195842 (+ 625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:25.196637 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:25.197652 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:25.197967 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:25.198024 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:25.200118 (+ 2094us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 21:08:25.200119 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 21:08:25.200119 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 21:08:25.200125 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 21:08:25.201565 (+ 1440us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:08:25.201914 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:08:25.201915 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:08:25.201916 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:08:25.201969 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:08:25.202162 (+ 193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:08:25.202163 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:08:25.202164 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:08:25.202271 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 21:08:25.202333 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0924 21:08:25.202496 (+ 163us) server_negotiation.cc:299] Negotiation successful
0924 21:08:25.202569 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260924 21:08:25.203935 7724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:25.195810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35220 (local address 127.0.98.60:39715)
0924 21:08:25.195844 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0924 21:08:25.195846 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 21:08:25.197731 (+ 1885us) server_negotiation.cc:407] Connection header received
0924 21:08:25.197755 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:08:25.197756 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:08:25.197764 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:08:25.197918 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:08:25.198543 (+ 625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:25.199207 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:25.200168 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:25.200431 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:25.200497 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:25.201520 (+ 1023us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 21:08:25.201521 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 21:08:25.201522 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 21:08:25.201527 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 21:08:25.202983 (+ 1456us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:08:25.203272 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:08:25.203273 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:08:25.203274 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:08:25.203320 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:08:25.203508 (+ 188us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:08:25.203508 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:08:25.203509 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:08:25.203603 (+ 94us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 21:08:25.203685 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0924 21:08:25.203771 (+ 86us) server_negotiation.cc:299] Negotiation successful
0924 21:08:25.203834 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260924 21:08:26.182642 7777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:26.182284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44697 (local address 127.0.98.2:40843)
0924 21:08:26.182441 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0924 21:08:26.182565 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":54,"threads_started":1}
I20260924 21:08:26.182984 7778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:26.182579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44697 (local address 127.0.98.1:38251)
0924 21:08:26.182837 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0924 21:08:26.182928 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":58,"threads_started":1}
W20260924 21:08:26.183037 7578 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:44697 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.62:44697: connect: Connection refused (error 111)
W20260924 21:08:26.183113 7442 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:44697 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.62:44697: connect: Connection refused (error 111)
I20260924 21:08:26.183624 7779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:26.183410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44697 (local address 127.0.98.3:44269)
0924 21:08:26.183526 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0924 21:08:26.183582 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":62,"threads_started":1}
W20260924 21:08:26.183758 7714 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:44697 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.62:44697: connect: Connection refused (error 111)
I20260924 21:08:26.685930 7780 raft_consensus.cc:482] T 00000000000000000000000000000000 P a5110fb5b32e4973948078ef72376369 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0f53f4b63d8e4da0905f8db79d1fc7db)
I20260924 21:08:26.686059 7781 raft_consensus.cc:482] T 00000000000000000000000000000000 P 65393c0abc864ec4b9d3d76877a9a863 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0f53f4b63d8e4da0905f8db79d1fc7db)
I20260924 21:08:26.686080 7780 raft_consensus.cc:504] T 00000000000000000000000000000000 P a5110fb5b32e4973948078ef72376369 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "0f53f4b63d8e4da0905f8db79d1fc7db" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44697 } } peers { permanent_uuid: "a5110fb5b32e4973948078ef72376369" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34067 } attrs { promote: false } } peers { permanent_uuid: "65393c0abc864ec4b9d3d76877a9a863" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39715 } attrs { promote: false } }
I20260924 21:08:26.686187 7781 raft_consensus.cc:504] T 00000000000000000000000000000000 P 65393c0abc864ec4b9d3d76877a9a863 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "0f53f4b63d8e4da0905f8db79d1fc7db" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44697 } } peers { permanent_uuid: "a5110fb5b32e4973948078ef72376369" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34067 } attrs { promote: false } } peers { permanent_uuid: "65393c0abc864ec4b9d3d76877a9a863" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39715 } attrs { promote: false } }
I20260924 21:08:26.686329 7780 leader_election.cc:288] T 00000000000000000000000000000000 P a5110fb5b32e4973948078ef72376369 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0f53f4b63d8e4da0905f8db79d1fc7db (127.0.98.62:44697), 65393c0abc864ec4b9d3d76877a9a863 (127.0.98.60:39715)
I20260924 21:08:26.686411 7781 leader_election.cc:288] T 00000000000000000000000000000000 P 65393c0abc864ec4b9d3d76877a9a863 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0f53f4b63d8e4da0905f8db79d1fc7db (127.0.98.62:44697), a5110fb5b32e4973948078ef72376369 (127.0.98.61:34067)
I20260924 21:08:26.686604 7782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:26.686403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44697 (local address 127.0.0.1:52808)
0924 21:08:26.686522 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0924 21:08:26.686564 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":42,"threads_started":1}
I20260924 21:08:26.686643 7784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:26.686474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44697 (local address 127.0.0.1:52812)
0924 21:08:26.686566 (+ 92us) negotiation.cc:107] Waiting for socket to connect
0924 21:08:26.686605 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79,"thread_start_us":38,"threads_started":1}
W20260924 21:08:26.686720 7061 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44697: connect: Connection refused (error 111)
W20260924 21:08:26.686733 7247 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44697: connect: Connection refused (error 111)
I20260924 21:08:26.687106 7782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:26.687014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44697 (local address 127.0.0.1:52816)
0924 21:08:26.687057 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0924 21:08:26.687083 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
I20260924 21:08:26.687146 7784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:26.687064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44697 (local address 127.0.0.1:52820)
0924 21:08:26.687100 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0924 21:08:26.687123 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
W20260924 21:08:26.687170 7061 leader_election.cc:334] T 00000000000000000000000000000000 P a5110fb5b32e4973948078ef72376369 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 0f53f4b63d8e4da0905f8db79d1fc7db (127.0.98.62:44697): Network error: Client connection negotiation failed: client connection to 127.0.98.62:44697: connect: Connection refused (error 111)
W20260924 21:08:26.687263 7247 leader_election.cc:334] T 00000000000000000000000000000000 P 65393c0abc864ec4b9d3d76877a9a863 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 0f53f4b63d8e4da0905f8db79d1fc7db (127.0.98.62:44697): Network error: Client connection negotiation failed: client connection to 127.0.98.62:44697: connect: Connection refused (error 111)
I20260924 21:08:26.689334 7789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:26.686852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48408 (local address 127.0.98.61:34067)
0924 21:08:26.686997 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0924 21:08:26.686999 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 21:08:26.687007 (+ 8us) server_negotiation.cc:407] Connection header received
0924 21:08:26.687029 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:08:26.687030 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:08:26.687039 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:08:26.687354 (+ 315us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 21:08:26.687385 (+ 31us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:26.687804 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:26.688605 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:26.688969 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:26.689056 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:26.689066 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0924 21:08:26.689212 (+ 146us) server_negotiation.cc:299] Negotiation successful
0924 21:08:26.689247 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99,"thread_start_us":87,"threads_started":1}
I20260924 21:08:26.689352 7785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:26.686581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:34067 (local address 127.0.0.1:48408)
0924 21:08:26.686748 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0924 21:08:26.686756 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0924 21:08:26.686800 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 21:08:26.687233 (+ 433us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 21:08:26.687234 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 21:08:26.687236 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 21:08:26.687295 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:26.687298 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:26.687889 (+ 591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:26.687890 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:26.688522 (+ 632us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:26.688523 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:26.689057 (+ 534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:26.689058 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:26.689130 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:26.689133 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 21:08:26.689205 (+ 72us) client_negotiation.cc:240] Negotiation successful
0924 21:08:26.689247 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":39,"threads_started":1}
I20260924 21:08:26.689404 7783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:26.686446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39715 (local address 127.0.0.1:35228)
0924 21:08:26.686590 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0924 21:08:26.686598 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0924 21:08:26.687176 (+ 578us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 21:08:26.687339 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 21:08:26.687339 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 21:08:26.687341 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 21:08:26.687385 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:26.687387 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:26.687916 (+ 529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:26.687917 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:26.688552 (+ 635us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:26.688553 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:26.689087 (+ 534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:26.689088 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:26.689151 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:26.689160 (+ 9us) client_negotiation.cc:769] Sending connection context
0924 21:08:26.689231 (+ 71us) client_negotiation.cc:240] Negotiation successful
0924 21:08:26.689275 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":39,"threads_started":1}
I20260924 21:08:26.689363 7786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:26.686637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35228 (local address 127.0.98.60:39715)
0924 21:08:26.686714 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0924 21:08:26.686716 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 21:08:26.686724 (+ 8us) server_negotiation.cc:407] Connection header received
0924 21:08:26.687263 (+ 539us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:08:26.687264 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:08:26.687273 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:08:26.687336 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 21:08:26.687436 (+ 100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:26.687846 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:26.688638 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:26.688998 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:26.689074 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:26.689085 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0924 21:08:26.689238 (+ 153us) server_negotiation.cc:299] Negotiation successful
0924 21:08:26.689275 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62,"thread_start_us":25,"threads_started":1}
I20260924 21:08:26.689756 7270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a5110fb5b32e4973948078ef72376369" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "65393c0abc864ec4b9d3d76877a9a863" is_pre_election: true
I20260924 21:08:26.689852 7084 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "65393c0abc864ec4b9d3d76877a9a863" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "a5110fb5b32e4973948078ef72376369" is_pre_election: true
I20260924 21:08:26.689903 7270 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 65393c0abc864ec4b9d3d76877a9a863 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a5110fb5b32e4973948078ef72376369 in term 1.
I20260924 21:08:26.689925 7084 raft_consensus.cc:2454] T 00000000000000000000000000000000 P a5110fb5b32e4973948078ef72376369 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65393c0abc864ec4b9d3d76877a9a863 in term 1.
I20260924 21:08:26.690064 7058 leader_election.cc:302] T 00000000000000000000000000000000 P a5110fb5b32e4973948078ef72376369 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 65393c0abc864ec4b9d3d76877a9a863, a5110fb5b32e4973948078ef72376369; no voters: 0f53f4b63d8e4da0905f8db79d1fc7db
I20260924 21:08:26.690076 7246 leader_election.cc:302] T 00000000000000000000000000000000 P 65393c0abc864ec4b9d3d76877a9a863 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 65393c0abc864ec4b9d3d76877a9a863, a5110fb5b32e4973948078ef72376369; no voters: 0f53f4b63d8e4da0905f8db79d1fc7db
I20260924 21:08:26.690162 7781 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 65393c0abc864ec4b9d3d76877a9a863 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260924 21:08:26.690196 7781 raft_consensus.cc:482] T 00000000000000000000000000000000 P 65393c0abc864ec4b9d3d76877a9a863 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 0f53f4b63d8e4da0905f8db79d1fc7db)
I20260924 21:08:26.690217 7781 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 65393c0abc864ec4b9d3d76877a9a863 [term 1 FOLLOWER]: Advancing to term 2
I20260924 21:08:26.690780 7781 raft_consensus.cc:504] T 00000000000000000000000000000000 P 65393c0abc864ec4b9d3d76877a9a863 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "0f53f4b63d8e4da0905f8db79d1fc7db" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44697 } } peers { permanent_uuid: "a5110fb5b32e4973948078ef72376369" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34067 } attrs { promote: false } } peers { permanent_uuid: "65393c0abc864ec4b9d3d76877a9a863" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39715 } attrs { promote: false } }
I20260924 21:08:26.690902 7781 leader_election.cc:288] T 00000000000000000000000000000000 P 65393c0abc864ec4b9d3d76877a9a863 [CANDIDATE]: Term 2 election: Requested vote from peers 0f53f4b63d8e4da0905f8db79d1fc7db (127.0.98.62:44697), a5110fb5b32e4973948078ef72376369 (127.0.98.61:34067)
I20260924 21:08:26.690968 7780 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a5110fb5b32e4973948078ef72376369 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260924 21:08:26.691006 7780 raft_consensus.cc:482] T 00000000000000000000000000000000 P a5110fb5b32e4973948078ef72376369 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 0f53f4b63d8e4da0905f8db79d1fc7db)
I20260924 21:08:26.691031 7780 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a5110fb5b32e4973948078ef72376369 [term 1 FOLLOWER]: Advancing to term 2
I20260924 21:08:26.691036 7785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:26.690960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44697 (local address 127.0.0.1:52832)
0924 21:08:26.690994 (+ 34us) negotiation.cc:107] Waiting for socket to connect
0924 21:08:26.691015 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
I20260924 21:08:26.691039 7084 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "65393c0abc864ec4b9d3d76877a9a863" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "a5110fb5b32e4973948078ef72376369"
I20260924 21:08:26.691282 7785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:26.691199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44697 (local address 127.0.0.1:52844)
0924 21:08:26.691246 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0924 21:08:26.691267 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260924 21:08:26.691404 7247 leader_election.cc:334] T 00000000000000000000000000000000 P 65393c0abc864ec4b9d3d76877a9a863 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 0f53f4b63d8e4da0905f8db79d1fc7db (127.0.98.62:44697): Network error: Client connection negotiation failed: client connection to 127.0.98.62:44697: connect: Connection refused (error 111)
I20260924 21:08:26.691711 7780 raft_consensus.cc:504] T 00000000000000000000000000000000 P a5110fb5b32e4973948078ef72376369 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "0f53f4b63d8e4da0905f8db79d1fc7db" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44697 } } peers { permanent_uuid: "a5110fb5b32e4973948078ef72376369" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34067 } attrs { promote: false } } peers { permanent_uuid: "65393c0abc864ec4b9d3d76877a9a863" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39715 } attrs { promote: false } }
I20260924 21:08:26.691821 7084 raft_consensus.cc:2379] T 00000000000000000000000000000000 P a5110fb5b32e4973948078ef72376369 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 65393c0abc864ec4b9d3d76877a9a863 in current term 2: Already voted for candidate a5110fb5b32e4973948078ef72376369 in this term.
I20260924 21:08:26.691948 7270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a5110fb5b32e4973948078ef72376369" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "65393c0abc864ec4b9d3d76877a9a863"
I20260924 21:08:26.691979 7246 leader_election.cc:302] T 00000000000000000000000000000000 P 65393c0abc864ec4b9d3d76877a9a863 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 65393c0abc864ec4b9d3d76877a9a863; no voters: 0f53f4b63d8e4da0905f8db79d1fc7db, a5110fb5b32e4973948078ef72376369
I20260924 21:08:26.691991 7783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:26.691892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44697 (local address 127.0.0.1:52860)
0924 21:08:26.691953 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0924 21:08:26.691971 (+ 18us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260924 21:08:26.692014 7270 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 65393c0abc864ec4b9d3d76877a9a863 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate a5110fb5b32e4973948078ef72376369 in current term 2: Already voted for candidate 65393c0abc864ec4b9d3d76877a9a863 in this term.
I20260924 21:08:26.691836 7780 leader_election.cc:288] T 00000000000000000000000000000000 P a5110fb5b32e4973948078ef72376369 [CANDIDATE]: Term 2 election: Requested vote from peers 0f53f4b63d8e4da0905f8db79d1fc7db (127.0.98.62:44697), 65393c0abc864ec4b9d3d76877a9a863 (127.0.98.60:39715)
I20260924 21:08:26.692059 7781 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 65393c0abc864ec4b9d3d76877a9a863 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260924 21:08:26.692204 7782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:26.692127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44697 (local address 127.0.0.1:52874)
0924 21:08:26.692159 (+ 32us) negotiation.cc:107] Waiting for socket to connect
0924 21:08:26.692185 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":17}
W20260924 21:08:26.692278 7061 leader_election.cc:334] T 00000000000000000000000000000000 P a5110fb5b32e4973948078ef72376369 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 0f53f4b63d8e4da0905f8db79d1fc7db (127.0.98.62:44697): Network error: Client connection negotiation failed: client connection to 127.0.98.62:44697: connect: Connection refused (error 111)
I20260924 21:08:26.692324 7061 leader_election.cc:302] T 00000000000000000000000000000000 P a5110fb5b32e4973948078ef72376369 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a5110fb5b32e4973948078ef72376369; no voters: 0f53f4b63d8e4da0905f8db79d1fc7db, 65393c0abc864ec4b9d3d76877a9a863
I20260924 21:08:26.692401 7780 raft_consensus.cc:2726] T 00000000000000000000000000000000 P a5110fb5b32e4973948078ef72376369 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260924 21:08:27.183683 7791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:27.183459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44697 (local address 127.0.98.1:43329)
0924 21:08:27.183602 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0924 21:08:27.183634 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":52,"threads_started":1}
I20260924 21:08:27.183833 7790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:27.183438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44697 (local address 127.0.98.2:53819)
0924 21:08:27.183561 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0924 21:08:27.183596 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":66,"threads_started":1}
I20260924 21:08:27.184191 7792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:27.184024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44697 (local address 127.0.98.3:57703)
0924 21:08:27.184123 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0924 21:08:27.184149 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86,"thread_start_us":53,"threads_started":1}
I20260924 21:08:28.184540 7794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:28.184257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44697 (local address 127.0.98.2:48995)
0924 21:08:28.184356 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0924 21:08:28.184491 (+ 135us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73,"thread_start_us":37,"threads_started":1}
I20260924 21:08:28.184542 7793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:28.184160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44697 (local address 127.0.98.1:36011)
0924 21:08:28.184351 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0924 21:08:28.184459 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":59,"threads_started":1}
I20260924 21:08:28.184796 7795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:28.184645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44697 (local address 127.0.98.3:58837)
0924 21:08:28.184738 (+ 93us) negotiation.cc:107] Waiting for socket to connect
0924 21:08:28.184762 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81,"thread_start_us":47,"threads_started":1}
I20260924 21:08:28.580566 7796 raft_consensus.cc:482] T 00000000000000000000000000000000 P 65393c0abc864ec4b9d3d76877a9a863 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 21:08:28.580636 7796 raft_consensus.cc:504] T 00000000000000000000000000000000 P 65393c0abc864ec4b9d3d76877a9a863 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "0f53f4b63d8e4da0905f8db79d1fc7db" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44697 } } peers { permanent_uuid: "a5110fb5b32e4973948078ef72376369" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34067 } attrs { promote: false } } peers { permanent_uuid: "65393c0abc864ec4b9d3d76877a9a863" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39715 } attrs { promote: false } }
I20260924 21:08:28.580754 7796 leader_election.cc:288] T 00000000000000000000000000000000 P 65393c0abc864ec4b9d3d76877a9a863 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0f53f4b63d8e4da0905f8db79d1fc7db (127.0.98.62:44697), a5110fb5b32e4973948078ef72376369 (127.0.98.61:34067)
I20260924 21:08:28.580955 7084 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "65393c0abc864ec4b9d3d76877a9a863" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "a5110fb5b32e4973948078ef72376369" is_pre_election: true
I20260924 21:08:28.581030 7084 raft_consensus.cc:2454] T 00000000000000000000000000000000 P a5110fb5b32e4973948078ef72376369 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65393c0abc864ec4b9d3d76877a9a863 in term 2.
I20260924 21:08:28.581045 7797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:28.580871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44697 (local address 127.0.0.1:53114)
0924 21:08:28.580961 (+ 90us) negotiation.cc:107] Waiting for socket to connect
0924 21:08:28.580996 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73,"thread_start_us":55,"threads_started":1}
I20260924 21:08:28.581171 7246 leader_election.cc:302] T 00000000000000000000000000000000 P 65393c0abc864ec4b9d3d76877a9a863 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65393c0abc864ec4b9d3d76877a9a863, a5110fb5b32e4973948078ef72376369; no voters:
I20260924 21:08:28.581247 7796 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 65393c0abc864ec4b9d3d76877a9a863 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260924 21:08:28.581296 7796 raft_consensus.cc:482] T 00000000000000000000000000000000 P 65393c0abc864ec4b9d3d76877a9a863 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 21:08:28.581315 7796 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 65393c0abc864ec4b9d3d76877a9a863 [term 2 FOLLOWER]: Advancing to term 3
I20260924 21:08:28.581478 7797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:28.581401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44697 (local address 127.0.0.1:53122)
0924 21:08:28.581430 (+ 29us) negotiation.cc:107] Waiting for socket to connect
0924 21:08:28.581452 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":13}
W20260924 21:08:28.581542 7247 leader_election.cc:334] T 00000000000000000000000000000000 P 65393c0abc864ec4b9d3d76877a9a863 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 0f53f4b63d8e4da0905f8db79d1fc7db (127.0.98.62:44697): Network error: Client connection negotiation failed: client connection to 127.0.98.62:44697: connect: Connection refused (error 111)
I20260924 21:08:28.582011 7796 raft_consensus.cc:504] T 00000000000000000000000000000000 P 65393c0abc864ec4b9d3d76877a9a863 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "0f53f4b63d8e4da0905f8db79d1fc7db" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44697 } } peers { permanent_uuid: "a5110fb5b32e4973948078ef72376369" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34067 } attrs { promote: false } } peers { permanent_uuid: "65393c0abc864ec4b9d3d76877a9a863" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39715 } attrs { promote: false } }
I20260924 21:08:28.582139 7796 leader_election.cc:288] T 00000000000000000000000000000000 P 65393c0abc864ec4b9d3d76877a9a863 [CANDIDATE]: Term 3 election: Requested vote from peers 0f53f4b63d8e4da0905f8db79d1fc7db (127.0.98.62:44697), a5110fb5b32e4973948078ef72376369 (127.0.98.61:34067)
I20260924 21:08:28.582278 7797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:28.582207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44697 (local address 127.0.0.1:53132)
0924 21:08:28.582241 (+ 34us) negotiation.cc:107] Waiting for socket to connect
0924 21:08:28.582264 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":17}
I20260924 21:08:28.582280 7084 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "65393c0abc864ec4b9d3d76877a9a863" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "a5110fb5b32e4973948078ef72376369"
I20260924 21:08:28.582376 7084 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a5110fb5b32e4973948078ef72376369 [term 2 FOLLOWER]: Advancing to term 3
I20260924 21:08:28.582567 7797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:28.582484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44697 (local address 127.0.0.1:53148)
0924 21:08:28.582529 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0924 21:08:28.582550 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260924 21:08:28.582640 7247 leader_election.cc:334] T 00000000000000000000000000000000 P 65393c0abc864ec4b9d3d76877a9a863 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 0f53f4b63d8e4da0905f8db79d1fc7db (127.0.98.62:44697): Network error: Client connection negotiation failed: client connection to 127.0.98.62:44697: connect: Connection refused (error 111)
I20260924 21:08:28.582998 7084 raft_consensus.cc:2454] T 00000000000000000000000000000000 P a5110fb5b32e4973948078ef72376369 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 65393c0abc864ec4b9d3d76877a9a863 in term 3.
I20260924 21:08:28.583151 7246 leader_election.cc:302] T 00000000000000000000000000000000 P 65393c0abc864ec4b9d3d76877a9a863 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 65393c0abc864ec4b9d3d76877a9a863, a5110fb5b32e4973948078ef72376369; no voters: 0f53f4b63d8e4da0905f8db79d1fc7db
I20260924 21:08:28.583225 7796 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 65393c0abc864ec4b9d3d76877a9a863 [term 3 FOLLOWER]: Leader election won for term 3
I20260924 21:08:28.583313 7796 raft_consensus.cc:686] T 00000000000000000000000000000000 P 65393c0abc864ec4b9d3d76877a9a863 [term 3 LEADER]: Becoming Leader. State: Replica: 65393c0abc864ec4b9d3d76877a9a863, State: Running, Role: LEADER
I20260924 21:08:28.583380 7796 consensus_queue.cc:237] T 00000000000000000000000000000000 P 65393c0abc864ec4b9d3d76877a9a863 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "0f53f4b63d8e4da0905f8db79d1fc7db" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44697 } } peers { permanent_uuid: "a5110fb5b32e4973948078ef72376369" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34067 } attrs { promote: false } } peers { permanent_uuid: "65393c0abc864ec4b9d3d76877a9a863" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39715 } attrs { promote: false } }
I20260924 21:08:28.583674 7799 sys_catalog.cc:455] T 00000000000000000000000000000000 P 65393c0abc864ec4b9d3d76877a9a863 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 65393c0abc864ec4b9d3d76877a9a863. Latest consensus state: current_term: 3 leader_uuid: "65393c0abc864ec4b9d3d76877a9a863" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "0f53f4b63d8e4da0905f8db79d1fc7db" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44697 } } peers { permanent_uuid: "a5110fb5b32e4973948078ef72376369" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34067 } attrs { promote: false } } peers { permanent_uuid: "65393c0abc864ec4b9d3d76877a9a863" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39715 } attrs { promote: false } } }
I20260924 21:08:28.583746 7799 sys_catalog.cc:458] T 00000000000000000000000000000000 P 65393c0abc864ec4b9d3d76877a9a863 [sys.catalog]: This master's current role is: LEADER
I20260924 21:08:28.583894 7801 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 21:08:28.584270 7801 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 21:08:28.584354 7801 catalog_manager.cc:1259] Loaded cluster ID: e2489afc08c64420b12bd07cdab5f107
I20260924 21:08:28.584383 7801 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 21:08:28.584545 7801 catalog_manager.cc:1504] Loading token signing keys...
I20260924 21:08:28.584628 7801 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 65393c0abc864ec4b9d3d76877a9a863: Loaded TSK: 0
I20260924 21:08:28.584756 7801 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 21:08:28.597123 7260 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35220:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 21:08:28.598539 7260 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 21:08:28.599406 7797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:28.599310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44697 (local address 127.0.0.1:53176)
0924 21:08:28.599355 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0924 21:08:28.599380 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260924 21:08:28.599427 7084 raft_consensus.cc:1261] T 00000000000000000000000000000000 P a5110fb5b32e4973948078ef72376369 [term 3 FOLLOWER]: Refusing update from remote peer 65393c0abc864ec4b9d3d76877a9a863: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20260924 21:08:28.599660 7796 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 65393c0abc864ec4b9d3d76877a9a863 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5110fb5b32e4973948078ef72376369" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34067 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260924 21:08:28.599721 7797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:28.599629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44697 (local address 127.0.0.1:53180)
0924 21:08:28.599675 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0924 21:08:28.599699 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260924 21:08:28.599817 7247 consensus_peers.cc:487] T 00000000000000000000000000000000 P 65393c0abc864ec4b9d3d76877a9a863 -> Peer 0f53f4b63d8e4da0905f8db79d1fc7db (127.0.98.62:44697): Couldn't send request to peer 0f53f4b63d8e4da0905f8db79d1fc7db. Status: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44697: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260924 21:08:28.600325 7803 sys_catalog.cc:455] T 00000000000000000000000000000000 P a5110fb5b32e4973948078ef72376369 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 65393c0abc864ec4b9d3d76877a9a863. Latest consensus state: current_term: 3 leader_uuid: "65393c0abc864ec4b9d3d76877a9a863" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "0f53f4b63d8e4da0905f8db79d1fc7db" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44697 } } peers { permanent_uuid: "a5110fb5b32e4973948078ef72376369" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34067 } attrs { promote: false } } peers { permanent_uuid: "65393c0abc864ec4b9d3d76877a9a863" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39715 } attrs { promote: false } } }
I20260924 21:08:28.600421 7803 sys_catalog.cc:458] T 00000000000000000000000000000000 P a5110fb5b32e4973948078ef72376369 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 21:08:28.600736 7796 sys_catalog.cc:455] T 00000000000000000000000000000000 P 65393c0abc864ec4b9d3d76877a9a863 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "65393c0abc864ec4b9d3d76877a9a863" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "0f53f4b63d8e4da0905f8db79d1fc7db" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44697 } } peers { permanent_uuid: "a5110fb5b32e4973948078ef72376369" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34067 } attrs { promote: false } } peers { permanent_uuid: "65393c0abc864ec4b9d3d76877a9a863" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39715 } attrs { promote: false } } }
I20260924 21:08:28.600805 7796 sys_catalog.cc:458] T 00000000000000000000000000000000 P 65393c0abc864ec4b9d3d76877a9a863 [sys.catalog]: This master's current role is: LEADER
I20260924 21:08:28.600823 7803 sys_catalog.cc:455] T 00000000000000000000000000000000 P a5110fb5b32e4973948078ef72376369 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "65393c0abc864ec4b9d3d76877a9a863" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "0f53f4b63d8e4da0905f8db79d1fc7db" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44697 } } peers { permanent_uuid: "a5110fb5b32e4973948078ef72376369" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34067 } attrs { promote: false } } peers { permanent_uuid: "65393c0abc864ec4b9d3d76877a9a863" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39715 } attrs { promote: false } } }
I20260924 21:08:28.600903 7803 sys_catalog.cc:458] T 00000000000000000000000000000000 P a5110fb5b32e4973948078ef72376369 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 21:08:28.604925 7809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:28.602239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42360 (local address 127.0.98.3:37191)
0924 21:08:28.602483 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0924 21:08:28.602485 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 21:08:28.602516 (+ 31us) server_negotiation.cc:407] Connection header received
0924 21:08:28.602538 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:08:28.602539 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:08:28.602547 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:08:28.602625 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 21:08:28.602789 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:28.603332 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:28.604180 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:28.604509 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:28.604595 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:28.604606 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0924 21:08:28.604829 (+ 223us) server_negotiation.cc:299] Negotiation successful
0924 21:08:28.604882 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":225,"thread_start_us":46,"threads_started":1}
I20260924 21:08:28.605064 7797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:28.602171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:37191 (local address 127.0.0.1:42360)
0924 21:08:28.602212 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0924 21:08:28.602221 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 21:08:28.602261 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 21:08:28.602630 (+ 369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 21:08:28.602631 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 21:08:28.602633 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 21:08:28.602676 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:28.602677 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:28.603410 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:28.603411 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:28.604103 (+ 692us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:28.604104 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:28.604640 (+ 536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:28.604640 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:28.604696 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:28.604701 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 21:08:28.604743 (+ 42us) client_negotiation.cc:240] Negotiation successful
0924 21:08:28.604782 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28,"mutex_wait_us":28}
I20260924 21:08:28.605397 7647 tablet_service.cc:1467] Processing CreateTablet for tablet 055a9f216a034e9abf1e4b273e0b0493 (DEFAULT_TABLE table=test-table [id=d976ac200aed46f080a5d306162726b5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 21:08:28.605515 7647 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 055a9f216a034e9abf1e4b273e0b0493. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 21:08:28.605899 7811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:28.602359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34392 (local address 127.0.98.2:46665)
0924 21:08:28.602464 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0924 21:08:28.602466 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 21:08:28.602474 (+ 8us) server_negotiation.cc:407] Connection header received
0924 21:08:28.602500 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:08:28.602501 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:08:28.602512 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:08:28.602656 (+ 144us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 21:08:28.603416 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:28.603990 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:28.605031 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:28.605626 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:28.605696 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:28.605706 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0924 21:08:28.605790 (+ 84us) server_negotiation.cc:299] Negotiation successful
0924 21:08:28.605837 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84,"thread_start_us":45,"threads_started":1}
I20260924 21:08:28.605911 7810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:28.602226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:46665 (local address 127.0.0.1:34392)
0924 21:08:28.602385 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0924 21:08:28.602393 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0924 21:08:28.602431 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 21:08:28.602653 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 21:08:28.602653 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 21:08:28.602654 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 21:08:28.602678 (+ 24us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:28.602679 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:28.604067 (+ 1388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:28.604068 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:28.604954 (+ 886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:28.604955 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:28.605700 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:28.605701 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:28.605739 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:28.605742 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 21:08:28.605778 (+ 36us) client_negotiation.cc:240] Negotiation successful
0924 21:08:28.605856 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":57,"threads_started":1}
I20260924 21:08:28.606151 7511 tablet_service.cc:1467] Processing CreateTablet for tablet 055a9f216a034e9abf1e4b273e0b0493 (DEFAULT_TABLE table=test-table [id=d976ac200aed46f080a5d306162726b5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 21:08:28.606256 7511 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 055a9f216a034e9abf1e4b273e0b0493. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 21:08:28.607016 7808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:28.602265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41965 (local address 127.0.0.1:46532)
0924 21:08:28.602686 (+ 421us) negotiation.cc:107] Waiting for socket to connect
0924 21:08:28.602694 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0924 21:08:28.602731 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 21:08:28.602960 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 21:08:28.602961 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 21:08:28.602963 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 21:08:28.602998 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:28.602999 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:28.603586 (+ 587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:28.603587 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:28.604517 (+ 930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:28.604519 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:28.605298 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:28.605299 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:28.606833 (+ 1534us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:28.606838 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 21:08:28.606901 (+ 63us) client_negotiation.cc:240] Negotiation successful
0924 21:08:28.606936 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":404,"thread_start_us":80,"threads_started":1}
I20260924 21:08:28.606856 7813 tablet_bootstrap.cc:492] T 055a9f216a034e9abf1e4b273e0b0493 P 54766fc07b09440682225ff08c6814db: Bootstrap starting.
I20260924 21:08:28.607184 7812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:28.602370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46532 (local address 127.0.98.1:41965)
0924 21:08:28.602806 (+ 436us) server_negotiation.cc:206] Beginning negotiation
0924 21:08:28.602808 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 21:08:28.602814 (+ 6us) server_negotiation.cc:407] Connection header received
0924 21:08:28.602837 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:08:28.602837 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:08:28.602847 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:08:28.602929 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 21:08:28.603054 (+ 125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:28.603512 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:28.604596 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:28.605140 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:28.605268 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:28.605279 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0924 21:08:28.607090 (+ 1811us) server_negotiation.cc:299] Negotiation successful
0924 21:08:28.607129 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":419,"thread_start_us":53,"threads_started":1}
I20260924 21:08:28.607355 7375 tablet_service.cc:1467] Processing CreateTablet for tablet 055a9f216a034e9abf1e4b273e0b0493 (DEFAULT_TABLE table=test-table [id=d976ac200aed46f080a5d306162726b5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 21:08:28.607476 7375 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 055a9f216a034e9abf1e4b273e0b0493. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 21:08:28.607523 7814 tablet_bootstrap.cc:492] T 055a9f216a034e9abf1e4b273e0b0493 P 80a0113d8614402db4980fda67be9b4f: Bootstrap starting.
I20260924 21:08:28.607796 7814 tablet_bootstrap.cc:654] T 055a9f216a034e9abf1e4b273e0b0493 P 80a0113d8614402db4980fda67be9b4f: Neither blocks nor log segments found. Creating new log.
I20260924 21:08:28.608307 7814 tablet_bootstrap.cc:492] T 055a9f216a034e9abf1e4b273e0b0493 P 80a0113d8614402db4980fda67be9b4f: No bootstrap required, opened a new log
I20260924 21:08:28.608371 7814 ts_tablet_manager.cc:1397] T 055a9f216a034e9abf1e4b273e0b0493 P 80a0113d8614402db4980fda67be9b4f: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260924 21:08:28.608466 7814 raft_consensus.cc:348] T 055a9f216a034e9abf1e4b273e0b0493 P 80a0113d8614402db4980fda67be9b4f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54766fc07b09440682225ff08c6814db" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37191 } } peers { permanent_uuid: "80a0113d8614402db4980fda67be9b4f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46665 } } peers { permanent_uuid: "1111a5950630489ba30bd8b925f7a15b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41965 } }
I20260924 21:08:28.608534 7814 raft_consensus.cc:374] T 055a9f216a034e9abf1e4b273e0b0493 P 80a0113d8614402db4980fda67be9b4f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 21:08:28.608559 7814 raft_consensus.cc:729] T 055a9f216a034e9abf1e4b273e0b0493 P 80a0113d8614402db4980fda67be9b4f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80a0113d8614402db4980fda67be9b4f, State: Initialized, Role: FOLLOWER
I20260924 21:08:28.608609 7814 consensus_queue.cc:260] T 055a9f216a034e9abf1e4b273e0b0493 P 80a0113d8614402db4980fda67be9b4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54766fc07b09440682225ff08c6814db" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37191 } } peers { permanent_uuid: "80a0113d8614402db4980fda67be9b4f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46665 } } peers { permanent_uuid: "1111a5950630489ba30bd8b925f7a15b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41965 } }
I20260924 21:08:28.608501 7816 tablet_bootstrap.cc:492] T 055a9f216a034e9abf1e4b273e0b0493 P 1111a5950630489ba30bd8b925f7a15b: Bootstrap starting.
I20260924 21:08:28.608695 7813 tablet_bootstrap.cc:654] T 055a9f216a034e9abf1e4b273e0b0493 P 54766fc07b09440682225ff08c6814db: Neither blocks nor log segments found. Creating new log.
I20260924 21:08:28.608779 7814 ts_tablet_manager.cc:1428] T 055a9f216a034e9abf1e4b273e0b0493 P 80a0113d8614402db4980fda67be9b4f: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260924 21:08:28.609055 7794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:28.608949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44697 (local address 127.0.98.2:46497)
0924 21:08:28.609000 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0924 21:08:28.609025 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":21}
I20260924 21:08:28.609287 7577 heartbeater.cc:503] Master 127.0.98.60:39715 was elected leader, sending a full tablet report...
I20260924 21:08:28.609455 7816 tablet_bootstrap.cc:654] T 055a9f216a034e9abf1e4b273e0b0493 P 1111a5950630489ba30bd8b925f7a15b: Neither blocks nor log segments found. Creating new log.
I20260924 21:08:28.609463 7794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:28.609381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44697 (local address 127.0.98.2:58311)
0924 21:08:28.609417 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0924 21:08:28.609439 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
I20260924 21:08:28.610085 7813 tablet_bootstrap.cc:492] T 055a9f216a034e9abf1e4b273e0b0493 P 54766fc07b09440682225ff08c6814db: No bootstrap required, opened a new log
I20260924 21:08:28.610127 7813 ts_tablet_manager.cc:1397] T 055a9f216a034e9abf1e4b273e0b0493 P 54766fc07b09440682225ff08c6814db: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260924 21:08:28.610140 7816 tablet_bootstrap.cc:492] T 055a9f216a034e9abf1e4b273e0b0493 P 1111a5950630489ba30bd8b925f7a15b: No bootstrap required, opened a new log
I20260924 21:08:28.610198 7816 ts_tablet_manager.cc:1397] T 055a9f216a034e9abf1e4b273e0b0493 P 1111a5950630489ba30bd8b925f7a15b: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 21:08:28.610280 7813 raft_consensus.cc:348] T 055a9f216a034e9abf1e4b273e0b0493 P 54766fc07b09440682225ff08c6814db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54766fc07b09440682225ff08c6814db" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37191 } } peers { permanent_uuid: "80a0113d8614402db4980fda67be9b4f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46665 } } peers { permanent_uuid: "1111a5950630489ba30bd8b925f7a15b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41965 } }
I20260924 21:08:28.610349 7813 raft_consensus.cc:374] T 055a9f216a034e9abf1e4b273e0b0493 P 54766fc07b09440682225ff08c6814db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 21:08:28.610363 7813 raft_consensus.cc:729] T 055a9f216a034e9abf1e4b273e0b0493 P 54766fc07b09440682225ff08c6814db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54766fc07b09440682225ff08c6814db, State: Initialized, Role: FOLLOWER
I20260924 21:08:28.610312 7816 raft_consensus.cc:348] T 055a9f216a034e9abf1e4b273e0b0493 P 1111a5950630489ba30bd8b925f7a15b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54766fc07b09440682225ff08c6814db" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37191 } } peers { permanent_uuid: "80a0113d8614402db4980fda67be9b4f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46665 } } peers { permanent_uuid: "1111a5950630489ba30bd8b925f7a15b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41965 } }
I20260924 21:08:28.610395 7816 raft_consensus.cc:374] T 055a9f216a034e9abf1e4b273e0b0493 P 1111a5950630489ba30bd8b925f7a15b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 21:08:28.610423 7816 raft_consensus.cc:729] T 055a9f216a034e9abf1e4b273e0b0493 P 1111a5950630489ba30bd8b925f7a15b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1111a5950630489ba30bd8b925f7a15b, State: Initialized, Role: FOLLOWER
I20260924 21:08:28.610427 7813 consensus_queue.cc:260] T 055a9f216a034e9abf1e4b273e0b0493 P 54766fc07b09440682225ff08c6814db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54766fc07b09440682225ff08c6814db" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37191 } } peers { permanent_uuid: "80a0113d8614402db4980fda67be9b4f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46665 } } peers { permanent_uuid: "1111a5950630489ba30bd8b925f7a15b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41965 } }
I20260924 21:08:28.610469 7816 consensus_queue.cc:260] T 055a9f216a034e9abf1e4b273e0b0493 P 1111a5950630489ba30bd8b925f7a15b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54766fc07b09440682225ff08c6814db" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37191 } } peers { permanent_uuid: "80a0113d8614402db4980fda67be9b4f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46665 } } peers { permanent_uuid: "1111a5950630489ba30bd8b925f7a15b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41965 } }
I20260924 21:08:28.610611 7816 ts_tablet_manager.cc:1428] T 055a9f216a034e9abf1e4b273e0b0493 P 1111a5950630489ba30bd8b925f7a15b: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260924 21:08:28.610639 7813 ts_tablet_manager.cc:1428] T 055a9f216a034e9abf1e4b273e0b0493 P 54766fc07b09440682225ff08c6814db: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260924 21:08:28.610954 7795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:28.610846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44697 (local address 127.0.98.3:47175)
0924 21:08:28.610899 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0924 21:08:28.610925 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260924 21:08:28.610993 7793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:28.610911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44697 (local address 127.0.98.1:54289)
0924 21:08:28.610943 (+ 32us) negotiation.cc:107] Waiting for socket to connect
0924 21:08:28.610968 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":17}
I20260924 21:08:28.611362 7713 heartbeater.cc:503] Master 127.0.98.60:39715 was elected leader, sending a full tablet report...
I20260924 21:08:28.611363 7441 heartbeater.cc:503] Master 127.0.98.60:39715 was elected leader, sending a full tablet report...
I20260924 21:08:28.611652 7793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:28.611566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44697 (local address 127.0.98.1:53901)
0924 21:08:28.611600 (+ 34us) negotiation.cc:107] Waiting for socket to connect
0924 21:08:28.611628 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":19}
I20260924 21:08:28.611862 7795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:28.611782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44697 (local address 127.0.98.3:47327)
0924 21:08:28.611815 (+ 33us) negotiation.cc:107] Waiting for socket to connect
0924 21:08:28.611842 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":19}
W20260924 21:08:28.719411 7716 tablet.cc:2369] T 055a9f216a034e9abf1e4b273e0b0493 P 54766fc07b09440682225ff08c6814db: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 21:08:28.725090 7444 tablet.cc:2369] T 055a9f216a034e9abf1e4b273e0b0493 P 1111a5950630489ba30bd8b925f7a15b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 21:08:28.731634 7820 raft_consensus.cc:482] T 055a9f216a034e9abf1e4b273e0b0493 P 54766fc07b09440682225ff08c6814db [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 21:08:28.731690 7820 raft_consensus.cc:504] T 055a9f216a034e9abf1e4b273e0b0493 P 54766fc07b09440682225ff08c6814db [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54766fc07b09440682225ff08c6814db" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37191 } } peers { permanent_uuid: "80a0113d8614402db4980fda67be9b4f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46665 } } peers { permanent_uuid: "1111a5950630489ba30bd8b925f7a15b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41965 } }
I20260924 21:08:28.731935 7820 leader_election.cc:288] T 055a9f216a034e9abf1e4b273e0b0493 P 54766fc07b09440682225ff08c6814db [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 80a0113d8614402db4980fda67be9b4f (127.0.98.2:46665), 1111a5950630489ba30bd8b925f7a15b (127.0.98.1:41965)
I20260924 21:08:28.734404 7795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:28.732098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:46665 (local address 127.0.98.3:45009)
0924 21:08:28.732132 (+ 34us) negotiation.cc:107] Waiting for socket to connect
0924 21:08:28.732143 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 21:08:28.732187 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 21:08:28.732338 (+ 151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 21:08:28.732339 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 21:08:28.732341 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 21:08:28.732401 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:28.732402 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:28.733001 (+ 599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:28.733002 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:28.733689 (+ 687us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:28.733691 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:28.734191 (+ 500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:28.734192 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:28.734257 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:28.734262 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 21:08:28.734308 (+ 46us) client_negotiation.cc:240] Negotiation successful
0924 21:08:28.734342 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19}
I20260924 21:08:28.734431 7811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:28.732157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:45009 (local address 127.0.98.2:46665)
0924 21:08:28.732199 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0924 21:08:28.732201 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 21:08:28.732210 (+ 9us) server_negotiation.cc:407] Connection header received
0924 21:08:28.732237 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:08:28.732238 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:08:28.732250 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:08:28.732316 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 21:08:28.732476 (+ 160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:28.732931 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:28.733774 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:28.734120 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:28.734172 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:28.734180 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0924 21:08:28.734328 (+ 148us) server_negotiation.cc:299] Negotiation successful
0924 21:08:28.734361 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260924 21:08:28.734525 7822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:28.732098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41965 (local address 127.0.98.3:53367)
0924 21:08:28.732281 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0924 21:08:28.732290 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 21:08:28.732340 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 21:08:28.732476 (+ 136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 21:08:28.732477 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 21:08:28.732479 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 21:08:28.732529 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:28.732530 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:28.733054 (+ 524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:28.733055 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:28.733698 (+ 643us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:28.733700 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:28.734265 (+ 565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:28.734266 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:28.734338 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:28.734342 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 21:08:28.734403 (+ 61us) client_negotiation.cc:240] Negotiation successful
0924 21:08:28.734437 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160,"mutex_wait_us":40,"thread_start_us":92,"threads_started":1}
I20260924 21:08:28.734530 7812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:28.732168 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:53367 (local address 127.0.98.1:41965)
0924 21:08:28.732201 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0924 21:08:28.732203 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 21:08:28.732371 (+ 168us) server_negotiation.cc:407] Connection header received
0924 21:08:28.732402 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:08:28.732402 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:08:28.732413 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:08:28.732461 (+ 48us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 21:08:28.732591 (+ 130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:28.732981 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:28.733838 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:28.734171 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:28.734236 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:28.734246 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0924 21:08:28.734425 (+ 179us) server_negotiation.cc:299] Negotiation successful
0924 21:08:28.734464 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260924 21:08:28.734589 7531 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "055a9f216a034e9abf1e4b273e0b0493" candidate_uuid: "54766fc07b09440682225ff08c6814db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80a0113d8614402db4980fda67be9b4f" is_pre_election: true
I20260924 21:08:28.734651 7531 raft_consensus.cc:2454] T 055a9f216a034e9abf1e4b273e0b0493 P 80a0113d8614402db4980fda67be9b4f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 54766fc07b09440682225ff08c6814db in term 0.
I20260924 21:08:28.734776 7395 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "055a9f216a034e9abf1e4b273e0b0493" candidate_uuid: "54766fc07b09440682225ff08c6814db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1111a5950630489ba30bd8b925f7a15b" is_pre_election: true
I20260924 21:08:28.734799 7602 leader_election.cc:302] T 055a9f216a034e9abf1e4b273e0b0493 P 54766fc07b09440682225ff08c6814db [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 54766fc07b09440682225ff08c6814db, 80a0113d8614402db4980fda67be9b4f; no voters:
I20260924 21:08:28.734951 7395 raft_consensus.cc:2454] T 055a9f216a034e9abf1e4b273e0b0493 P 1111a5950630489ba30bd8b925f7a15b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 54766fc07b09440682225ff08c6814db in term 0.
I20260924 21:08:28.734993 7820 raft_consensus.cc:2781] T 055a9f216a034e9abf1e4b273e0b0493 P 54766fc07b09440682225ff08c6814db [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 21:08:28.735036 7820 raft_consensus.cc:482] T 055a9f216a034e9abf1e4b273e0b0493 P 54766fc07b09440682225ff08c6814db [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 21:08:28.735069 7820 raft_consensus.cc:3037] T 055a9f216a034e9abf1e4b273e0b0493 P 54766fc07b09440682225ff08c6814db [term 0 FOLLOWER]: Advancing to term 1
I20260924 21:08:28.735694 7820 raft_consensus.cc:504] T 055a9f216a034e9abf1e4b273e0b0493 P 54766fc07b09440682225ff08c6814db [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54766fc07b09440682225ff08c6814db" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37191 } } peers { permanent_uuid: "80a0113d8614402db4980fda67be9b4f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46665 } } peers { permanent_uuid: "1111a5950630489ba30bd8b925f7a15b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41965 } }
I20260924 21:08:28.735822 7820 leader_election.cc:288] T 055a9f216a034e9abf1e4b273e0b0493 P 54766fc07b09440682225ff08c6814db [CANDIDATE]: Term 1 election: Requested vote from peers 80a0113d8614402db4980fda67be9b4f (127.0.98.2:46665), 1111a5950630489ba30bd8b925f7a15b (127.0.98.1:41965)
I20260924 21:08:28.735965 7531 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "055a9f216a034e9abf1e4b273e0b0493" candidate_uuid: "54766fc07b09440682225ff08c6814db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80a0113d8614402db4980fda67be9b4f"
I20260924 21:08:28.735981 7395 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "055a9f216a034e9abf1e4b273e0b0493" candidate_uuid: "54766fc07b09440682225ff08c6814db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1111a5950630489ba30bd8b925f7a15b"
I20260924 21:08:28.736054 7395 raft_consensus.cc:3037] T 055a9f216a034e9abf1e4b273e0b0493 P 1111a5950630489ba30bd8b925f7a15b [term 0 FOLLOWER]: Advancing to term 1
I20260924 21:08:28.736079 7531 raft_consensus.cc:3037] T 055a9f216a034e9abf1e4b273e0b0493 P 80a0113d8614402db4980fda67be9b4f [term 0 FOLLOWER]: Advancing to term 1
I20260924 21:08:28.736610 7531 raft_consensus.cc:2454] T 055a9f216a034e9abf1e4b273e0b0493 P 80a0113d8614402db4980fda67be9b4f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 54766fc07b09440682225ff08c6814db in term 1.
I20260924 21:08:28.736660 7395 raft_consensus.cc:2454] T 055a9f216a034e9abf1e4b273e0b0493 P 1111a5950630489ba30bd8b925f7a15b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 54766fc07b09440682225ff08c6814db in term 1.
I20260924 21:08:28.736768 7602 leader_election.cc:302] T 055a9f216a034e9abf1e4b273e0b0493 P 54766fc07b09440682225ff08c6814db [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 54766fc07b09440682225ff08c6814db, 80a0113d8614402db4980fda67be9b4f; no voters:
I20260924 21:08:28.736866 7820 raft_consensus.cc:2781] T 055a9f216a034e9abf1e4b273e0b0493 P 54766fc07b09440682225ff08c6814db [term 1 FOLLOWER]: Leader election won for term 1
I20260924 21:08:28.736999 7820 raft_consensus.cc:686] T 055a9f216a034e9abf1e4b273e0b0493 P 54766fc07b09440682225ff08c6814db [term 1 LEADER]: Becoming Leader. State: Replica: 54766fc07b09440682225ff08c6814db, State: Running, Role: LEADER
I20260924 21:08:28.737082 7820 consensus_queue.cc:237] T 055a9f216a034e9abf1e4b273e0b0493 P 54766fc07b09440682225ff08c6814db [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54766fc07b09440682225ff08c6814db" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37191 } } peers { permanent_uuid: "80a0113d8614402db4980fda67be9b4f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46665 } } peers { permanent_uuid: "1111a5950630489ba30bd8b925f7a15b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41965 } }
I20260924 21:08:28.737262 7822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:28.737185 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44697 (local address 127.0.98.3:47621)
0924 21:08:28.737216 (+ 31us) negotiation.cc:107] Waiting for socket to connect
0924 21:08:28.737245 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":15}
I20260924 21:08:28.737679 7259 catalog_manager.cc:5495] T 055a9f216a034e9abf1e4b273e0b0493 P 54766fc07b09440682225ff08c6814db reported cstate change: term changed from 0 to 1, leader changed from <none> to 54766fc07b09440682225ff08c6814db (127.0.98.3). New cstate: current_term: 1 leader_uuid: "54766fc07b09440682225ff08c6814db" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54766fc07b09440682225ff08c6814db" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37191 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "80a0113d8614402db4980fda67be9b4f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46665 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1111a5950630489ba30bd8b925f7a15b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41965 } health_report { overall_health: UNKNOWN } } }
I20260924 21:08:28.748049 7809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:28.745843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42376 (local address 127.0.98.3:37191)
0924 21:08:28.745880 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0924 21:08:28.745881 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 21:08:28.745888 (+ 7us) server_negotiation.cc:407] Connection header received
0924 21:08:28.745946 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:08:28.745947 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:08:28.745963 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:08:28.746019 (+ 56us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 21:08:28.746100 (+ 81us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:28.746581 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:28.747285 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:28.747486 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:28.747589 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:28.747744 (+ 155us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 21:08:28.747848 (+ 104us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 21:08:28.747927 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0924 21:08:28.747967 (+ 40us) server_negotiation.cc:299] Negotiation successful
0924 21:08:28.747984 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260924 21:08:28.748847 7531 raft_consensus.cc:1261] T 055a9f216a034e9abf1e4b273e0b0493 P 80a0113d8614402db4980fda67be9b4f [term 1 FOLLOWER]: Refusing update from remote peer 54766fc07b09440682225ff08c6814db: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 21:08:28.748886 7395 raft_consensus.cc:1261] T 055a9f216a034e9abf1e4b273e0b0493 P 1111a5950630489ba30bd8b925f7a15b [term 1 FOLLOWER]: Refusing update from remote peer 54766fc07b09440682225ff08c6814db: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 21:08:28.749192 7820 consensus_queue.cc:1035] T 055a9f216a034e9abf1e4b273e0b0493 P 54766fc07b09440682225ff08c6814db [LEADER]: Connected to new peer: Peer: permanent_uuid: "80a0113d8614402db4980fda67be9b4f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46665 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 21:08:28.749328 7823 consensus_queue.cc:1035] T 055a9f216a034e9abf1e4b273e0b0493 P 54766fc07b09440682225ff08c6814db [LEADER]: Connected to new peer: Peer: permanent_uuid: "1111a5950630489ba30bd8b925f7a15b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 21:08:28.749959 7794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:28.749873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44697 (local address 127.0.98.2:49803)
0924 21:08:28.749904 (+ 31us) negotiation.cc:107] Waiting for socket to connect
0924 21:08:28.749934 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":16}
I20260924 21:08:28.750313 7793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:28.750214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44697 (local address 127.0.98.1:60315)
0924 21:08:28.750261 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0924 21:08:28.750289 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260924 21:08:28.750770 7826 mvcc.cc:205] Tried to move back new op lower bound from 7333003709433606144 to 7333003709387620352. Current Snapshot: MvccSnapshot[applied={T|T < 7333003709433606144}]
I20260924 21:08:28.751183 7822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:28.750999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44697 (local address 127.0.98.3:51153)
0924 21:08:28.751134 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0924 21:08:28.751161 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122}
I20260924 21:08:28.751443 7794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:28.751368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44697 (local address 127.0.98.2:38171)
0924 21:08:28.751399 (+ 31us) negotiation.cc:107] Waiting for socket to connect
0924 21:08:28.751418 (+ 19us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
I20260924 21:08:28.751730 7793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:28.751649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44697 (local address 127.0.98.1:51245)
0924 21:08:28.751684 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0924 21:08:28.751705 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":20}
I20260924 21:08:28.751801 7822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:28.751713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44697 (local address 127.0.98.3:59573)
0924 21:08:28.751743 (+ 30us) negotiation.cc:107] Waiting for socket to connect
0924 21:08:28.751776 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":14}
I20260924 21:08:28.752678 7259 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35220:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 21:08:28.752753 7259 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35220:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 21:08:28.753846 7259 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 65393c0abc864ec4b9d3d76877a9a863: Sending DeleteTablet for 3 replicas of tablet 055a9f216a034e9abf1e4b273e0b0493
I20260924 21:08:28.754099 7647 tablet_service.cc:1514] Processing DeleteTablet for tablet 055a9f216a034e9abf1e4b273e0b0493 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 21:08:28 UTC) from {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:42360
I20260924 21:08:28.754223 7511 tablet_service.cc:1514] Processing DeleteTablet for tablet 055a9f216a034e9abf1e4b273e0b0493 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 21:08:28 UTC) from {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:34392
I20260924 21:08:28.754240 7375 tablet_service.cc:1514] Processing DeleteTablet for tablet 055a9f216a034e9abf1e4b273e0b0493 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 21:08:28 UTC) from {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:46532
I20260924 21:08:28.754439 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAzNNjS/build/release/bin/kudu with pid 7312
I20260924 21:08:28.755209 7836 tablet_replica.cc:330] T 055a9f216a034e9abf1e4b273e0b0493 P 54766fc07b09440682225ff08c6814db: stopping tablet replica
I20260924 21:08:28.755254 7836 raft_consensus.cc:2229] T 055a9f216a034e9abf1e4b273e0b0493 P 54766fc07b09440682225ff08c6814db [term 1 LEADER]: Raft consensus shutting down.
I20260924 21:08:28.755308 7836 raft_consensus.cc:2258] T 055a9f216a034e9abf1e4b273e0b0493 P 54766fc07b09440682225ff08c6814db [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 21:08:28.755565 7836 ts_tablet_manager.cc:1905] T 055a9f216a034e9abf1e4b273e0b0493 P 54766fc07b09440682225ff08c6814db: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 21:08:28.756927 7836 ts_tablet_manager.cc:1918] T 055a9f216a034e9abf1e4b273e0b0493 P 54766fc07b09440682225ff08c6814db: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 21:08:28.756984 7836 log.cc:1196] T 055a9f216a034e9abf1e4b273e0b0493 P 54766fc07b09440682225ff08c6814db: Deleting WAL directory at /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790284076800039-392-0/minicluster-data/ts-2/wal/wals/055a9f216a034e9abf1e4b273e0b0493
I20260924 21:08:28.757205 7836 ts_tablet_manager.cc:1939] T 055a9f216a034e9abf1e4b273e0b0493 P 54766fc07b09440682225ff08c6814db: Deleting consensus metadata
I20260924 21:08:28.757474 7244 catalog_manager.cc:4841] TS 54766fc07b09440682225ff08c6814db (127.0.98.3:37191): tablet 055a9f216a034e9abf1e4b273e0b0493 (table test-table [id=d976ac200aed46f080a5d306162726b5]) successfully deleted
I20260924 21:08:28.760768 7837 tablet_replica.cc:330] T 055a9f216a034e9abf1e4b273e0b0493 P 80a0113d8614402db4980fda67be9b4f: stopping tablet replica
I20260924 21:08:28.760813 7808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:28.760708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41965 (local address 127.0.0.1:46548)
0924 21:08:28.760761 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0924 21:08:28.760788 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:41965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260924 21:08:28.760882 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAzNNjS/build/release/bin/kudu with pid 7448
W20260924 21:08:28.760918 7244 catalog_manager.cc:4584] TS 1111a5950630489ba30bd8b925f7a15b (127.0.98.1:41965): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 055a9f216a034e9abf1e4b273e0b0493: Network error: Client connection negotiation failed: client connection to 127.0.98.1:41965: connect: Connection refused (error 111)
I20260924 21:08:28.765363 7808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:28.765251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:46665 (local address 127.0.0.1:34406)
0924 21:08:28.765292 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0924 21:08:28.765333 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:46665: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
I20260924 21:08:28.765517 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAzNNjS/build/release/bin/kudu with pid 7584
I20260924 21:08:28.769973 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAzNNjS/build/release/bin/kudu with pid 6940
I20260924 21:08:28.774145 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAzNNjS/build/release/bin/kudu with pid 7128
2026-09-24T21:08:28Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (6380 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790284076800039-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 21:08:28 dist-test-slave-c7tk krb5kdc[7841](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 21:08:28 dist-test-slave-c7tk krb5kdc[7841](info): set up 2 sockets
Sep 24 21:08:28 dist-test-slave-c7tk krb5kdc[7841](info): commencing operation
krb5kdc: starting...
W20260924 21:08:29.689999 392 mini_kdc.cc:121] Time spent starting KDC: real 0.895s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790284076800039-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790284076800039-392-0/krb5kdc/test-user.keytab.
Sep 24 21:08:29 dist-test-slave-c7tk krb5kdc[7841](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284109, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T21:08:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T21:08:29Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790284076800039-392-0/krb5kdc/oryx_127.0.98.62.keytab.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790284076800039-392-0/krb5kdc/oryx_127.0.98.62.keytab.
I20260924 21:08:29.786309 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAzNNjS/build/release/bin/kudu
/tmp/dist-test-taskAzNNjS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790284076800039-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790284076800039-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790284076800039-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790284076800039-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42643
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790284076800039-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45937
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42643
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790284076800039-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790284076800039-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790284076800039-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 21:08:29.856896 7854 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 21:08:29.857080 7854 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 21:08:29.857110 7854 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 21:08:29.858150 7854 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 21:08:29.858206 7854 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260924 21:08:29.858225 7854 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 21:08:29.858245 7854 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 21:08:29.858273 7854 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 21:08:29.858294 7854 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 21:08:29.859637 7854 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45937
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790284076800039-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790284076800039-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42643
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790284076800039-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42643
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790284076800039-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7854
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790284076800039-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 19:49:06 UTC on ccycloud.quasar-vhqjoc.root.comops.site
build id 4670864
I20260924 21:08:29.859881 7854 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 21:08:29.860085 7854 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 21:08:29.864337 7859 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 21:08:29.864398 7860 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 21:08:29.864482 7854 server_base.cc:1034] running on GCE node
W20260924 21:08:29.864585 7862 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 21:08:29.864781 7854 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 21:08:29.865031 7854 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 21:08:29.866173 7854 hybrid_clock.cc:648] HybridClock initialized: now 1790284109866157 us; error 27 us; skew 500 ppm
Sep 24 21:08:29 dist-test-slave-c7tk krb5kdc[7841](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284109, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 21:08:29.868266 7854 init.cc:377] Logged in from keytab as oryx/127.0.98.62@KRBTEST.COM (short username oryx)
I20260924 21:08:29.868624 7854 webserver.cc:460] Webserver started at http://127.0.98.62:44287/ using document root <none> and password file <none>
I20260924 21:08:29.868825 7854 fs_manager.cc:359] Metadata directory not provided
I20260924 21:08:29.868888 7854 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 21:08:29.869017 7854 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 21:08:29.870050 7854 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790284076800039-392-0/minicluster-data/master-0/data/instance:
uuid: "6a3f903b8ae5424eb578b88c276a8ac9"
format_stamp: "Formatted at 2026-09-24 21:08:29 on dist-test-slave-c7tk"
server_key: "84525da1f5a1edbd8bad10e8dc2f09c7"
server_key_iv: "4ca2e105686d97dd34098979ed5e3557"
server_key_version: "encryptionkey@0"
I20260924 21:08:29.870395 7854 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790284076800039-392-0/minicluster-data/master-0/wal/instance:
uuid: "6a3f903b8ae5424eb578b88c276a8ac9"
format_stamp: "Formatted at 2026-09-24 21:08:29 on dist-test-slave-c7tk"
server_key: "84525da1f5a1edbd8bad10e8dc2f09c7"
server_key_iv: "4ca2e105686d97dd34098979ed5e3557"
server_key_version: "encryptionkey@0"
I20260924 21:08:29.871508 7854 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 21:08:29.872079 7869 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 21:08:29.872217 7854 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 21:08:29.872298 7854 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790284076800039-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790284076800039-392-0/minicluster-data/master-0/wal
uuid: "6a3f903b8ae5424eb578b88c276a8ac9"
format_stamp: "Formatted at 2026-09-24 21:08:29 on dist-test-slave-c7tk"
server_key: "84525da1f5a1edbd8bad10e8dc2f09c7"
server_key_iv: "4ca2e105686d97dd34098979ed5e3557"
server_key_version: "encryptionkey@0"
I20260924 21:08:29.872371 7854 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790284076800039-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790284076800039-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790284076800039-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 21:08:29.890522 7854 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 21:08:29.891956 7854 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 21:08:29.892089 7854 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 21:08:29.896059 7854 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42643
I20260924 21:08:29.896096 7931 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42643 every 8 connection(s)
I20260924 21:08:29.896341 7854 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790284076800039-392-0/minicluster-data/master-0/data/info.pb
I20260924 21:08:29.896801 7932 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 21:08:29.898622 7932 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6a3f903b8ae5424eb578b88c276a8ac9: Bootstrap starting.
I20260924 21:08:29.899115 7932 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6a3f903b8ae5424eb578b88c276a8ac9: Neither blocks nor log segments found. Creating new log.
I20260924 21:08:29.899426 7932 log.cc:824] T 00000000000000000000000000000000 P 6a3f903b8ae5424eb578b88c276a8ac9: Log is configured to *not* fsync() on all Append() calls
I20260924 21:08:29.899994 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAzNNjS/build/release/bin/kudu as pid 7854
I20260924 21:08:29.900079 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790284076800039-392-0/minicluster-data/master-0/wal/instance
I20260924 21:08:29.900154 7932 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6a3f903b8ae5424eb578b88c276a8ac9: No bootstrap required, opened a new log
I20260924 21:08:29.900275 392 external_mini_cluster.cc:1428] Setting key ae78778bdf8bc797a1873ac2f60523ed
I20260924 21:08:29.902204 7932 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6a3f903b8ae5424eb578b88c276a8ac9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a3f903b8ae5424eb578b88c276a8ac9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42643 } }
I20260924 21:08:29.902307 7932 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6a3f903b8ae5424eb578b88c276a8ac9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 21:08:29.902325 7932 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6a3f903b8ae5424eb578b88c276a8ac9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a3f903b8ae5424eb578b88c276a8ac9, State: Initialized, Role: FOLLOWER
I20260924 21:08:29.902483 7932 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6a3f903b8ae5424eb578b88c276a8ac9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a3f903b8ae5424eb578b88c276a8ac9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42643 } }
I20260924 21:08:29.902560 7932 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6a3f903b8ae5424eb578b88c276a8ac9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 21:08:29.902599 7932 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6a3f903b8ae5424eb578b88c276a8ac9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 21:08:29.902663 7932 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6a3f903b8ae5424eb578b88c276a8ac9 [term 0 FOLLOWER]: Advancing to term 1
I20260924 21:08:29.903118 7932 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6a3f903b8ae5424eb578b88c276a8ac9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a3f903b8ae5424eb578b88c276a8ac9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42643 } }
I20260924 21:08:29.903219 7932 leader_election.cc:302] T 00000000000000000000000000000000 P 6a3f903b8ae5424eb578b88c276a8ac9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6a3f903b8ae5424eb578b88c276a8ac9; no voters:
I20260924 21:08:29.903424 7932 leader_election.cc:288] T 00000000000000000000000000000000 P 6a3f903b8ae5424eb578b88c276a8ac9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 21:08:29.903455 7937 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6a3f903b8ae5424eb578b88c276a8ac9 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 21:08:29.903558 7937 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6a3f903b8ae5424eb578b88c276a8ac9 [term 1 LEADER]: Becoming Leader. State: Replica: 6a3f903b8ae5424eb578b88c276a8ac9, State: Running, Role: LEADER
I20260924 21:08:29.903646 7932 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6a3f903b8ae5424eb578b88c276a8ac9 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 21:08:29.903622 7937 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6a3f903b8ae5424eb578b88c276a8ac9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a3f903b8ae5424eb578b88c276a8ac9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42643 } }
I20260924 21:08:29.903880 7937 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6a3f903b8ae5424eb578b88c276a8ac9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6a3f903b8ae5424eb578b88c276a8ac9. Latest consensus state: current_term: 1 leader_uuid: "6a3f903b8ae5424eb578b88c276a8ac9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a3f903b8ae5424eb578b88c276a8ac9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42643 } } }
I20260924 21:08:29.903896 7939 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6a3f903b8ae5424eb578b88c276a8ac9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6a3f903b8ae5424eb578b88c276a8ac9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a3f903b8ae5424eb578b88c276a8ac9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42643 } } }
I20260924 21:08:29.903949 7937 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6a3f903b8ae5424eb578b88c276a8ac9 [sys.catalog]: This master's current role is: LEADER
I20260924 21:08:29.903954 7939 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6a3f903b8ae5424eb578b88c276a8ac9 [sys.catalog]: This master's current role is: LEADER
I20260924 21:08:29.904103 7941 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 21:08:29.904487 7941 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 21:08:29.906075 7941 catalog_manager.cc:1347] Generated new cluster ID: e5c2016cdaf6420194e7bd2fab20f6f8
I20260924 21:08:29.906123 7941 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 24 21:08:29 dist-test-slave-c7tk krb5kdc[7841](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284109, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260924 21:08:29.911386 7936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:29.900781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40712 (local address 127.0.98.62:42643)
0924 21:08:29.900952 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0924 21:08:29.900960 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 21:08:29.900987 (+ 27us) server_negotiation.cc:407] Connection header received
0924 21:08:29.902193 (+ 1206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:08:29.902198 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:08:29.902239 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:08:29.902375 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:08:29.903106 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:29.904367 (+ 1261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:29.905241 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:29.905706 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:29.905788 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:29.908301 (+ 2513us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 21:08:29.908305 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 21:08:29.908305 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 21:08:29.908316 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 21:08:29.909996 (+ 1680us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:08:29.910371 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:08:29.910373 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:08:29.910374 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:08:29.910471 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:08:29.910649 (+ 178us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:08:29.910649 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:08:29.910650 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:08:29.910847 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 21:08:29.911011 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0924 21:08:29.911066 (+ 55us) server_negotiation.cc:299] Negotiation successful
0924 21:08:29.911190 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":78,"threads_started":1}
I20260924 21:08:29.923030 7941 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 21:08:29.923571 7941 catalog_manager.cc:1504] Loading token signing keys...
I20260924 21:08:29.931931 7941 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6a3f903b8ae5424eb578b88c276a8ac9: Generated new TSK 0
I20260924 21:08:29.932060 7941 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790284076800039-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790284076800039-392-0/krb5kdc/oryx_127.0.98.1.keytab.
I20260924 21:08:29.960299 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAzNNjS/build/release/bin/kudu
/tmp/dist-test-taskAzNNjS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790284076800039-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790284076800039-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790284076800039-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790284076800039-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42643
--keytab_file=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790284076800039-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45937
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790284076800039-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790284076800039-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790284076800039-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 21:08:30.030386 7958 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 21:08:30.030578 7958 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 21:08:30.030612 7958 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 21:08:30.031626 7958 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 21:08:30.031684 7958 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 21:08:30.031715 7958 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 21:08:30.031752 7958 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 21:08:30.033008 7958 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45937
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790284076800039-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790284076800039-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790284076800039-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790284076800039-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:42643
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7958
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790284076800039-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 19:49:06 UTC on ccycloud.quasar-vhqjoc.root.comops.site
build id 4670864
I20260924 21:08:30.033229 7958 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 21:08:30.033419 7958 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 21:08:30.037604 7963 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 21:08:30.037628 7964 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 21:08:30.037657 7958 server_base.cc:1034] running on GCE node
W20260924 21:08:30.037773 7966 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 21:08:30.037968 7958 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 21:08:30.038185 7958 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 21:08:30.039348 7958 hybrid_clock.cc:648] HybridClock initialized: now 1790284110039327 us; error 50 us; skew 500 ppm
Sep 24 21:08:30 dist-test-slave-c7tk krb5kdc[7841](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284110, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 21:08:30.042412 7958 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260924 21:08:30.042876 7958 webserver.cc:460] Webserver started at http://127.0.98.1:33193/ using document root <none> and password file <none>
I20260924 21:08:30.043088 7958 fs_manager.cc:359] Metadata directory not provided
I20260924 21:08:30.043138 7958 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 21:08:30.043252 7958 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 21:08:30.044399 7958 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790284076800039-392-0/minicluster-data/ts-0/data/instance:
uuid: "efc3778361914d90b0e2c86fa65e639b"
format_stamp: "Formatted at 2026-09-24 21:08:30 on dist-test-slave-c7tk"
server_key: "2883e46ecef2096024d41e66d94ed547"
server_key_iv: "eb950deb1eaa33ab043f117de10a6ab0"
server_key_version: "encryptionkey@0"
I20260924 21:08:30.044741 7958 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790284076800039-392-0/minicluster-data/ts-0/wal/instance:
uuid: "efc3778361914d90b0e2c86fa65e639b"
format_stamp: "Formatted at 2026-09-24 21:08:30 on dist-test-slave-c7tk"
server_key: "2883e46ecef2096024d41e66d94ed547"
server_key_iv: "eb950deb1eaa33ab043f117de10a6ab0"
server_key_version: "encryptionkey@0"
I20260924 21:08:30.045934 7958 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 21:08:30.046876 7973 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 21:08:30.047042 7958 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 21:08:30.047123 7958 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790284076800039-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790284076800039-392-0/minicluster-data/ts-0/wal
uuid: "efc3778361914d90b0e2c86fa65e639b"
format_stamp: "Formatted at 2026-09-24 21:08:30 on dist-test-slave-c7tk"
server_key: "2883e46ecef2096024d41e66d94ed547"
server_key_iv: "eb950deb1eaa33ab043f117de10a6ab0"
server_key_version: "encryptionkey@0"
I20260924 21:08:30.047190 7958 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790284076800039-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790284076800039-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790284076800039-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 21:08:30.064855 7958 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 21:08:30.066679 7958 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 21:08:30.066819 7958 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 21:08:30.067219 7958 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 21:08:30.067276 7958 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 21:08:30.067301 7958 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 21:08:30.067332 7958 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 21:08:30.072758 7958 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:45765
I20260924 21:08:30.073037 7958 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790284076800039-392-0/minicluster-data/ts-0/data/info.pb
Sep 24 21:08:30 dist-test-slave-c7tk krb5kdc[7841](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284110, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260924 21:08:30.073477 8088 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:45765 every 8 connection(s)
I20260924 21:08:30.074168 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAzNNjS/build/release/bin/kudu as pid 7958
I20260924 21:08:30.074256 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790284076800039-392-0/minicluster-data/ts-0/wal/instance
I20260924 21:08:30.074445 392 external_mini_cluster.cc:1428] Setting key 02a9ce44e4d8234a0efe344cf364ff6d
I20260924 21:08:30.076483 7936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:30.067556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:50955 (local address 127.0.98.62:42643)
0924 21:08:30.067615 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0924 21:08:30.067617 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 21:08:30.067824 (+ 207us) server_negotiation.cc:407] Connection header received
0924 21:08:30.068232 (+ 408us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:08:30.068234 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:08:30.068247 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:08:30.068314 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:08:30.069048 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:30.069913 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:30.071326 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:30.071625 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:30.071703 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:30.073731 (+ 2028us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 21:08:30.073733 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 21:08:30.073734 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 21:08:30.073741 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 21:08:30.075199 (+ 1458us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:08:30.075628 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:08:30.075629 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:08:30.075629 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:08:30.075683 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:08:30.075909 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:08:30.075910 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:08:30.075910 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:08:30.076008 (+ 98us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 21:08:30.076068 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0924 21:08:30.076304 (+ 236us) server_negotiation.cc:299] Negotiation successful
0924 21:08:30.076394 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260924 21:08:30.076566 7982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:30.067536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42643 (local address 127.0.98.1:50955)
0924 21:08:30.067737 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0924 21:08:30.067760 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0924 21:08:30.068144 (+ 384us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 21:08:30.068402 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 21:08:30.068405 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 21:08:30.068732 (+ 327us) client_negotiation.cc:189] Negotiated authn=SASL
0924 21:08:30.068955 (+ 223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:30.068960 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:30.070002 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:30.070003 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:30.071224 (+ 1221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:30.071226 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:30.071717 (+ 491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:30.071717 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:30.071849 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:30.072282 (+ 433us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 21:08:30.072291 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 21:08:30.073623 (+ 1332us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 21:08:30.075269 (+ 1646us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 21:08:30.075274 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 21:08:30.075276 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 21:08:30.075559 (+ 283us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 21:08:30.075749 (+ 190us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 21:08:30.075750 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 21:08:30.075750 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 21:08:30.075846 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 21:08:30.076070 (+ 224us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 21:08:30.076075 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 21:08:30.076188 (+ 113us) client_negotiation.cc:769] Sending connection context
0924 21:08:30.076271 (+ 83us) client_negotiation.cc:240] Negotiation successful
0924 21:08:30.076395 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":77,"threads_started":1}
I20260924 21:08:30.076977 8089 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42643
I20260924 21:08:30.077085 8089 heartbeater.cc:464] Registering TS with master...
I20260924 21:08:30.077273 8089 heartbeater.cc:511] Master 127.0.98.62:42643 requested a full tablet report, sending...
I20260924 21:08:30.077728 7885 ts_manager.cc:194] Registered new tserver with Master: efc3778361914d90b0e2c86fa65e639b (127.0.98.1:45765)
I20260924 21:08:30.078575 7885 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:50955
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260924 21:08:30.083962 7936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:30.078794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:39503 (local address 127.0.98.62:42643)
0924 21:08:30.078828 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0924 21:08:30.078839 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 21:08:30.078847 (+ 8us) server_negotiation.cc:407] Connection header received
0924 21:08:30.078866 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:08:30.078866 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:08:30.078875 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:08:30.078933 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:08:30.079231 (+ 298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:30.079863 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:30.080902 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:30.081056 (+ 154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:30.081105 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:30.081989 (+ 884us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 21:08:30.081990 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 21:08:30.081991 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 21:08:30.081995 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 21:08:30.083168 (+ 1173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:08:30.083428 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:08:30.083429 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:08:30.083429 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:08:30.083468 (+ 39us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:08:30.083673 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:08:30.083674 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:08:30.083674 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:08:30.083748 (+ 74us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 21:08:30.083791 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0924 21:08:30.083871 (+ 80us) server_negotiation.cc:299] Negotiation successful
0924 21:08:30.083917 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260924 21:08:30.083971 8096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:30.077764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42643 (local address 127.0.98.1:39503)
0924 21:08:30.077880 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0924 21:08:30.077888 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0924 21:08:30.077928 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 21:08:30.078946 (+ 1018us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 21:08:30.078946 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 21:08:30.079094 (+ 148us) client_negotiation.cc:189] Negotiated authn=SASL
0924 21:08:30.079157 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:30.079158 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:30.079961 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:30.079962 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:30.080826 (+ 864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:30.080828 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:30.081127 (+ 299us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:30.081128 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:30.081186 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:30.081525 (+ 339us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 21:08:30.081530 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 21:08:30.081911 (+ 381us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 21:08:30.083227 (+ 1316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 21:08:30.083228 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 21:08:30.083228 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 21:08:30.083373 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 21:08:30.083562 (+ 189us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 21:08:30.083563 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 21:08:30.083563 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 21:08:30.083621 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 21:08:30.083787 (+ 166us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 21:08:30.083788 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 21:08:30.083813 (+ 25us) client_negotiation.cc:769] Sending connection context
0924 21:08:30.083858 (+ 45us) client_negotiation.cc:240] Negotiation successful
0924 21:08:30.083916 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":76,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790284076800039-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790284076800039-392-0/krb5kdc/oryx_127.0.98.2.keytab.
I20260924 21:08:30.097590 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAzNNjS/build/release/bin/kudu
/tmp/dist-test-taskAzNNjS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790284076800039-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790284076800039-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790284076800039-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790284076800039-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42643
--keytab_file=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790284076800039-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45937
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790284076800039-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790284076800039-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790284076800039-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 21:08:30.167335 8098 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 21:08:30.167484 8098 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 21:08:30.167505 8098 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 21:08:30.168437 8098 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 21:08:30.168464 8098 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 21:08:30.168480 8098 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 21:08:30.168494 8098 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 21:08:30.169744 8098 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45937
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790284076800039-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790284076800039-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790284076800039-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790284076800039-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:42643
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8098
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790284076800039-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 19:49:06 UTC on ccycloud.quasar-vhqjoc.root.comops.site
build id 4670864
I20260924 21:08:30.169976 8098 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 21:08:30.170142 8098 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 21:08:30.174188 8106 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 21:08:30.174217 8104 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 21:08:30.174257 8098 server_base.cc:1034] running on GCE node
W20260924 21:08:30.174198 8103 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 21:08:30.174480 8098 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 21:08:30.174679 8098 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 21:08:30.175822 8098 hybrid_clock.cc:648] HybridClock initialized: now 1790284110175805 us; error 40 us; skew 500 ppm
Sep 24 21:08:30 dist-test-slave-c7tk krb5kdc[7841](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284110, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 21:08:30.177855 8098 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260924 21:08:30.178205 8098 webserver.cc:460] Webserver started at http://127.0.98.2:38833/ using document root <none> and password file <none>
I20260924 21:08:30.178397 8098 fs_manager.cc:359] Metadata directory not provided
I20260924 21:08:30.178434 8098 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 21:08:30.178574 8098 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 21:08:30.179491 8098 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790284076800039-392-0/minicluster-data/ts-1/data/instance:
uuid: "1b3ebea12b784499ad16419ad850345a"
format_stamp: "Formatted at 2026-09-24 21:08:30 on dist-test-slave-c7tk"
server_key: "b5e94699d5c1bf8993aa175025a8dae9"
server_key_iv: "214d0990b2a05dfed127a10e0de55e3e"
server_key_version: "encryptionkey@0"
I20260924 21:08:30.179816 8098 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790284076800039-392-0/minicluster-data/ts-1/wal/instance:
uuid: "1b3ebea12b784499ad16419ad850345a"
format_stamp: "Formatted at 2026-09-24 21:08:30 on dist-test-slave-c7tk"
server_key: "b5e94699d5c1bf8993aa175025a8dae9"
server_key_iv: "214d0990b2a05dfed127a10e0de55e3e"
server_key_version: "encryptionkey@0"
I20260924 21:08:30.180987 8098 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 21:08:30.181578 8113 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 21:08:30.181720 8098 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 21:08:30.181785 8098 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790284076800039-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790284076800039-392-0/minicluster-data/ts-1/wal
uuid: "1b3ebea12b784499ad16419ad850345a"
format_stamp: "Formatted at 2026-09-24 21:08:30 on dist-test-slave-c7tk"
server_key: "b5e94699d5c1bf8993aa175025a8dae9"
server_key_iv: "214d0990b2a05dfed127a10e0de55e3e"
server_key_version: "encryptionkey@0"
I20260924 21:08:30.181880 8098 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790284076800039-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790284076800039-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790284076800039-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 21:08:30.199082 8098 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 21:08:30.201035 8098 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 21:08:30.201119 8098 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 21:08:30.201457 8098 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 21:08:30.201493 8098 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 21:08:30.201531 8098 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 21:08:30.201555 8098 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 21:08:30.207562 8098 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:37145
I20260924 21:08:30.207617 8228 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:37145 every 8 connection(s)
Sep 24 21:08:30 dist-test-slave-c7tk krb5kdc[7841](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284110, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260924 21:08:30.207875 8098 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790284076800039-392-0/minicluster-data/ts-1/data/info.pb
I20260924 21:08:30.210425 7936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:30.201842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:54259 (local address 127.0.98.62:42643)
0924 21:08:30.201903 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0924 21:08:30.201905 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 21:08:30.202091 (+ 186us) server_negotiation.cc:407] Connection header received
0924 21:08:30.202660 (+ 569us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:08:30.202661 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:08:30.202671 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:08:30.202728 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:08:30.203455 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:30.204148 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:30.205585 (+ 1437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:30.205869 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:30.205935 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:30.208141 (+ 2206us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 21:08:30.208143 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 21:08:30.208143 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 21:08:30.208149 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 21:08:30.209289 (+ 1140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:08:30.209624 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:08:30.209625 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:08:30.209625 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:08:30.209695 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:08:30.209965 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:08:30.209966 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:08:30.209966 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:08:30.210063 (+ 97us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 21:08:30.210117 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0924 21:08:30.210305 (+ 188us) server_negotiation.cc:299] Negotiation successful
0924 21:08:30.210372 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260924 21:08:30.210569 8122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:30.201836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42643 (local address 127.0.98.2:54259)
0924 21:08:30.201970 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0924 21:08:30.202013 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 21:08:30.202558 (+ 545us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 21:08:30.202787 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 21:08:30.202790 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 21:08:30.203111 (+ 321us) client_negotiation.cc:189] Negotiated authn=SASL
0924 21:08:30.203361 (+ 250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:30.203366 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:30.204225 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:30.204226 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:30.205485 (+ 1259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:30.205488 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:30.205937 (+ 449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:30.205938 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:30.206065 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:30.206520 (+ 455us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 21:08:30.206529 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 21:08:30.208049 (+ 1520us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 21:08:30.209362 (+ 1313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 21:08:30.209367 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 21:08:30.209368 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 21:08:30.209560 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 21:08:30.209759 (+ 199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 21:08:30.209759 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 21:08:30.209760 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 21:08:30.209907 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 21:08:30.210112 (+ 205us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 21:08:30.210114 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 21:08:30.210205 (+ 91us) client_negotiation.cc:769] Sending connection context
0924 21:08:30.210274 (+ 69us) client_negotiation.cc:240] Negotiation successful
0924 21:08:30.210392 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":94,"thread_start_us":65,"threads_started":1}
I20260924 21:08:30.210955 8229 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42643
I20260924 21:08:30.211030 8229 heartbeater.cc:464] Registering TS with master...
I20260924 21:08:30.211176 8229 heartbeater.cc:511] Master 127.0.98.62:42643 requested a full tablet report, sending...
I20260924 21:08:30.211202 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAzNNjS/build/release/bin/kudu as pid 8098
I20260924 21:08:30.211287 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790284076800039-392-0/minicluster-data/ts-1/wal/instance
I20260924 21:08:30.211457 7886 ts_manager.cc:194] Registered new tserver with Master: 1b3ebea12b784499ad16419ad850345a (127.0.98.2:37145)
I20260924 21:08:30.211505 392 external_mini_cluster.cc:1428] Setting key 9fc36cb3ffeb95a3b9803d7a0f82f0c3
I20260924 21:08:30.211895 7886 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:54259
I20260924 21:08:30.217734 8235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:30.211818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42643 (local address 127.0.98.2:49105)
0924 21:08:30.211934 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0924 21:08:30.211943 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 21:08:30.211989 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 21:08:30.213062 (+ 1073us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 21:08:30.213063 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 21:08:30.213179 (+ 116us) client_negotiation.cc:189] Negotiated authn=SASL
0924 21:08:30.213218 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:30.213219 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:30.214031 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:30.214031 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:30.214479 (+ 448us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:30.214480 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:30.214906 (+ 426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:30.214907 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:30.214958 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:30.215308 (+ 350us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 21:08:30.215311 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0924 21:08:30.215516 (+ 205us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 21:08:30.216952 (+ 1436us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 21:08:30.216952 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 21:08:30.216953 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 21:08:30.217081 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 21:08:30.217249 (+ 168us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 21:08:30.217250 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 21:08:30.217250 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 21:08:30.217289 (+ 39us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 21:08:30.217558 (+ 269us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 21:08:30.217558 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 21:08:30.217584 (+ 26us) client_negotiation.cc:769] Sending connection context
0924 21:08:30.217626 (+ 42us) client_negotiation.cc:240] Negotiation successful
0924 21:08:30.217681 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":60,"threads_started":1}
I20260924 21:08:30.217890 7936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:30.211943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:49105 (local address 127.0.98.62:42643)
0924 21:08:30.211970 (+ 27us) server_negotiation.cc:206] Beginning negotiation
0924 21:08:30.211972 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 21:08:30.211979 (+ 7us) server_negotiation.cc:407] Connection header received
0924 21:08:30.212039 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:08:30.212040 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:08:30.212047 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:08:30.212096 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:08:30.213291 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:30.213958 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:30.214552 (+ 594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:30.214831 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:30.214897 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:30.215597 (+ 700us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 21:08:30.215598 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 21:08:30.215599 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 21:08:30.215605 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 21:08:30.216876 (+ 1271us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:08:30.217139 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:08:30.217140 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:08:30.217140 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:08:30.217190 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:08:30.217331 (+ 141us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:08:30.217331 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:08:30.217331 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:08:30.217495 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 21:08:30.217558 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0924 21:08:30.217648 (+ 90us) server_negotiation.cc:299] Negotiation successful
0924 21:08:30.217707 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14}
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790284076800039-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790284076800039-392-0/krb5kdc/oryx_127.0.98.3.keytab.
I20260924 21:08:30.233327 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAzNNjS/build/release/bin/kudu
/tmp/dist-test-taskAzNNjS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790284076800039-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790284076800039-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790284076800039-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790284076800039-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42643
--keytab_file=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790284076800039-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45937
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790284076800039-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790284076800039-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790284076800039-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 21:08:30.303867 8238 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 21:08:30.304018 8238 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 21:08:30.304039 8238 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 21:08:30.305156 8238 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 21:08:30.305226 8238 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 21:08:30.305262 8238 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 21:08:30.305285 8238 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 21:08:30.306530 8238 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45937
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790284076800039-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790284076800039-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790284076800039-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790284076800039-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:42643
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8238
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790284076800039-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 19:49:06 UTC on ccycloud.quasar-vhqjoc.root.comops.site
build id 4670864
I20260924 21:08:30.306674 8238 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 21:08:30.306802 8238 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 21:08:30.311120 8246 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 21:08:30.311137 8243 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 21:08:30.311164 8238 server_base.cc:1034] running on GCE node
W20260924 21:08:30.311148 8244 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 21:08:30.311453 8238 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 21:08:30.311712 8238 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 21:08:30.312866 8238 hybrid_clock.cc:648] HybridClock initialized: now 1790284110312839 us; error 29 us; skew 500 ppm
Sep 24 21:08:30 dist-test-slave-c7tk krb5kdc[7841](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284110, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 21:08:30.314986 8238 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260924 21:08:30.315366 8238 webserver.cc:460] Webserver started at http://127.0.98.3:37015/ using document root <none> and password file <none>
I20260924 21:08:30.315526 8238 fs_manager.cc:359] Metadata directory not provided
I20260924 21:08:30.315560 8238 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 21:08:30.315634 8238 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 21:08:30.316443 8238 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790284076800039-392-0/minicluster-data/ts-2/data/instance:
uuid: "8a234a50319146b69dfea62fbce5654b"
format_stamp: "Formatted at 2026-09-24 21:08:30 on dist-test-slave-c7tk"
server_key: "6f8105b676ee96e96d361d4005beb3ad"
server_key_iv: "1bff78507c2cbd67f093d520d9c8192c"
server_key_version: "encryptionkey@0"
I20260924 21:08:30.316717 8238 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790284076800039-392-0/minicluster-data/ts-2/wal/instance:
uuid: "8a234a50319146b69dfea62fbce5654b"
format_stamp: "Formatted at 2026-09-24 21:08:30 on dist-test-slave-c7tk"
server_key: "6f8105b676ee96e96d361d4005beb3ad"
server_key_iv: "1bff78507c2cbd67f093d520d9c8192c"
server_key_version: "encryptionkey@0"
I20260924 21:08:30.317826 8238 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260924 21:08:30.318392 8253 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 21:08:30.318521 8238 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 21:08:30.318590 8238 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790284076800039-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790284076800039-392-0/minicluster-data/ts-2/wal
uuid: "8a234a50319146b69dfea62fbce5654b"
format_stamp: "Formatted at 2026-09-24 21:08:30 on dist-test-slave-c7tk"
server_key: "6f8105b676ee96e96d361d4005beb3ad"
server_key_iv: "1bff78507c2cbd67f093d520d9c8192c"
server_key_version: "encryptionkey@0"
I20260924 21:08:30.318634 8238 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790284076800039-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790284076800039-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790284076800039-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 21:08:30.328449 8238 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 21:08:30.330056 8238 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 21:08:30.330158 8238 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 21:08:30.330546 8238 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 21:08:30.330582 8238 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 21:08:30.330641 8238 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 21:08:30.330675 8238 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 21:08:30.337164 8238 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:35619
I20260924 21:08:30.337275 8368 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:35619 every 8 connection(s)
Sep 24 21:08:30 dist-test-slave-c7tk krb5kdc[7841](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284110, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260924 21:08:30.337551 8238 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790284076800039-392-0/minicluster-data/ts-2/data/info.pb
I20260924 21:08:30.340256 7936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:30.330976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:59599 (local address 127.0.98.62:42643)
0924 21:08:30.331037 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0924 21:08:30.331039 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 21:08:30.331255 (+ 216us) server_negotiation.cc:407] Connection header received
0924 21:08:30.331690 (+ 435us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:08:30.331691 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:08:30.331701 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:08:30.331773 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:08:30.332617 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:30.333399 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:30.335127 (+ 1728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:30.335438 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:30.335492 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:30.338133 (+ 2641us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 21:08:30.338135 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 21:08:30.338135 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 21:08:30.338141 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 21:08:30.339215 (+ 1074us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:08:30.339557 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:08:30.339558 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:08:30.339558 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:08:30.339605 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:08:30.339799 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:08:30.339799 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:08:30.339800 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:08:30.339893 (+ 93us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 21:08:30.339941 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0924 21:08:30.340139 (+ 198us) server_negotiation.cc:299] Negotiation successful
0924 21:08:30.340199 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260924 21:08:30.340399 8262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:30.330952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42643 (local address 127.0.98.3:59599)
0924 21:08:30.331166 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0924 21:08:30.331190 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0924 21:08:30.331596 (+ 406us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 21:08:30.331881 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 21:08:30.331885 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 21:08:30.332251 (+ 366us) client_negotiation.cc:189] Negotiated authn=SASL
0924 21:08:30.332522 (+ 271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:30.332527 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:30.333496 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:30.333497 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:30.335007 (+ 1510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:30.335010 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:30.335542 (+ 532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:30.335543 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:30.335701 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:30.336262 (+ 561us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 21:08:30.336273 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 21:08:30.338045 (+ 1772us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 21:08:30.339278 (+ 1233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 21:08:30.339283 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 21:08:30.339284 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 21:08:30.339492 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 21:08:30.339660 (+ 168us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 21:08:30.339661 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 21:08:30.339661 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 21:08:30.339748 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 21:08:30.339945 (+ 197us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 21:08:30.339948 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 21:08:30.340044 (+ 96us) client_negotiation.cc:769] Sending connection context
0924 21:08:30.340124 (+ 80us) client_negotiation.cc:240] Negotiation successful
0924 21:08:30.340227 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":81,"threads_started":1}
I20260924 21:08:30.340814 8369 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42643
I20260924 21:08:30.340896 8369 heartbeater.cc:464] Registering TS with master...
I20260924 21:08:30.341056 8369 heartbeater.cc:511] Master 127.0.98.62:42643 requested a full tablet report, sending...
I20260924 21:08:30.341415 7885 ts_manager.cc:194] Registered new tserver with Master: 8a234a50319146b69dfea62fbce5654b (127.0.98.3:35619)
I20260924 21:08:30.341796 7885 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:59599
I20260924 21:08:30.346639 7936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:30.341887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:51005 (local address 127.0.98.62:42643)
0924 21:08:30.341933 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0924 21:08:30.341935 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 21:08:30.342010 (+ 75us) server_negotiation.cc:407] Connection header received
0924 21:08:30.342070 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:08:30.342071 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:08:30.342086 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:08:30.342151 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:08:30.342457 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:30.342915 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:30.343571 (+ 656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:30.343767 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:30.343836 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:30.344531 (+ 695us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 21:08:30.344533 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 21:08:30.344533 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 21:08:30.344539 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 21:08:30.345612 (+ 1073us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:08:30.345987 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:08:30.345988 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:08:30.345988 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:08:30.346034 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:08:30.346219 (+ 185us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:08:30.346219 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:08:30.346220 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:08:30.346311 (+ 91us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 21:08:30.346370 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0924 21:08:30.346505 (+ 135us) server_negotiation.cc:299] Negotiation successful
0924 21:08:30.346566 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260924 21:08:30.346652 8375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:30.341818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42643 (local address 127.0.98.3:51005)
0924 21:08:30.341956 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0924 21:08:30.341966 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 21:08:30.342022 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 21:08:30.342157 (+ 135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 21:08:30.342158 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 21:08:30.342330 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0924 21:08:30.342374 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:30.342375 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:30.343007 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:30.343008 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:30.343490 (+ 482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:30.343491 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:30.343836 (+ 345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:30.343837 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:30.343896 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:30.344226 (+ 330us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 21:08:30.344229 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0924 21:08:30.344454 (+ 225us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 21:08:30.345699 (+ 1245us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 21:08:30.345700 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 21:08:30.345701 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 21:08:30.345914 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 21:08:30.346110 (+ 196us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 21:08:30.346110 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 21:08:30.346111 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 21:08:30.346158 (+ 47us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 21:08:30.346375 (+ 217us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 21:08:30.346375 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 21:08:30.346416 (+ 41us) client_negotiation.cc:769] Sending connection context
0924 21:08:30.346486 (+ 70us) client_negotiation.cc:240] Negotiation successful
0924 21:08:30.346566 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":79,"threads_started":1}
I20260924 21:08:30.346661 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAzNNjS/build/release/bin/kudu as pid 8238
I20260924 21:08:30.346762 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790284076800039-392-0/minicluster-data/ts-2/wal/instance
I20260924 21:08:30.346956 392 external_mini_cluster.cc:1428] Setting key 45ab2f9c5cc4bcc3471c376a2f949987
I20260924 21:08:30.347883 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 24 21:08:30 dist-test-slave-c7tk krb5kdc[7841](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284110, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 24 21:08:30 dist-test-slave-c7tk krb5kdc[7841](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 21:08:30 dist-test-slave-c7tk krb5kdc[7841](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260924 21:08:30.366710 7936 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 21:08:30.362512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40728 (local address 127.0.98.62:42643)
0924 21:08:30.362557 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0924 21:08:30.362558 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 21:08:30.362580 (+ 22us) server_negotiation.cc:407] Connection header received
0924 21:08:30.362647 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:08:30.362648 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:08:30.362655 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:08:30.362688 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:08:30.363146 (+ 458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:30.363542 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:30.364135 (+ 593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:30.364279 (+ 144us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:30.364319 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:30.366639 (+ 2320us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40728: BlockingRecv error: recv got EOF from 127.0.0.1:40728 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
Sep 24 21:08:30 dist-test-slave-c7tk krb5kdc[7841](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284110, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260924 21:08:30.372978 7936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:30.367314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40744 (local address 127.0.98.62:42643)
0924 21:08:30.367366 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0924 21:08:30.367368 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 21:08:30.367403 (+ 35us) server_negotiation.cc:407] Connection header received
0924 21:08:30.367468 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:08:30.367469 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:08:30.367476 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:08:30.367525 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:08:30.367901 (+ 376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:30.368295 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:30.368885 (+ 590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:30.369055 (+ 170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:30.369153 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:30.370940 (+ 1787us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 21:08:30.370941 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 21:08:30.370942 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 21:08:30.370948 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 21:08:30.372039 (+ 1091us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:08:30.372333 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:08:30.372334 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:08:30.372335 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:08:30.372383 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:08:30.372557 (+ 174us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:08:30.372558 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:08:30.372558 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:08:30.372658 (+ 100us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 21:08:30.372746 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0924 21:08:30.372836 (+ 90us) server_negotiation.cc:299] Negotiation successful
0924 21:08:30.372901 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260924 21:08:30.374409 7885 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40744:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 21:08:30.375881 7885 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 21:08:30.381410 8390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:30.377727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56618 (local address 127.0.98.3:35619)
0924 21:08:30.377864 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0924 21:08:30.377869 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 21:08:30.378013 (+ 144us) server_negotiation.cc:407] Connection header received
0924 21:08:30.378121 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:08:30.378123 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:08:30.378138 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:08:30.378185 (+ 47us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 21:08:30.378423 (+ 238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:30.379202 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:30.380536 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:30.381047 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:30.381076 (+ 29us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:30.381118 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0924 21:08:30.381298 (+ 180us) server_negotiation.cc:299] Negotiation successful
0924 21:08:30.381328 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103,"thread_start_us":92,"threads_started":1}
I20260924 21:08:30.381472 8392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:30.377680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:37145 (local address 127.0.0.1:35536)
0924 21:08:30.377941 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0924 21:08:30.377977 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0924 21:08:30.378025 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 21:08:30.378289 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 21:08:30.378290 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 21:08:30.378292 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 21:08:30.378361 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:30.378363 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:30.379588 (+ 1225us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:30.379589 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:30.380321 (+ 732us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:30.380323 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:30.381233 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:30.381234 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:30.381290 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:30.381294 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 21:08:30.381321 (+ 27us) client_negotiation.cc:240] Negotiation successful
0924 21:08:30.381356 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":246,"mutex_wait_us":1,"thread_start_us":52,"threads_started":1}
I20260924 21:08:30.381443 8389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:30.377631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:35619 (local address 127.0.0.1:56618)
0924 21:08:30.377892 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0924 21:08:30.377927 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0924 21:08:30.378036 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 21:08:30.378262 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 21:08:30.378264 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 21:08:30.378266 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 21:08:30.378343 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:30.378345 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:30.379286 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:30.379287 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:30.380445 (+ 1158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:30.380448 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:30.381159 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:30.381159 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:30.381205 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:30.381212 (+ 7us) client_negotiation.cc:769] Sending connection context
0924 21:08:30.381284 (+ 72us) client_negotiation.cc:240] Negotiation successful
0924 21:08:30.381328 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":241,"mutex_wait_us":22,"thread_start_us":60,"threads_started":1}
I20260924 21:08:30.381865 8391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:30.377785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35536 (local address 127.0.98.2:37145)
0924 21:08:30.378149 (+ 364us) server_negotiation.cc:206] Beginning negotiation
0924 21:08:30.378153 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 21:08:30.378163 (+ 10us) server_negotiation.cc:407] Connection header received
0924 21:08:30.378194 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:08:30.378196 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:08:30.378208 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:08:30.378278 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 21:08:30.378450 (+ 172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:30.379497 (+ 1047us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:30.380473 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:30.381150 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:30.381624 (+ 474us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:30.381698 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0924 21:08:30.381720 (+ 22us) server_negotiation.cc:299] Negotiation successful
0924 21:08:30.381753 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":333,"thread_start_us":127,"threads_started":1}
I20260924 21:08:30.381878 8393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:30.378005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41618 (local address 127.0.98.1:45765)
0924 21:08:30.378162 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0924 21:08:30.378166 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 21:08:30.378174 (+ 8us) server_negotiation.cc:407] Connection header received
0924 21:08:30.378208 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:08:30.378214 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:08:30.378224 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:08:30.378304 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 21:08:30.378518 (+ 214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:30.379578 (+ 1060us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:30.380764 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:30.381399 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:30.381455 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:30.381517 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0924 21:08:30.381693 (+ 176us) server_negotiation.cc:299] Negotiation successful
0924 21:08:30.381729 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":69,"threads_started":1}
I20260924 21:08:30.381754 8388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:30.377849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45765 (local address 127.0.0.1:41618)
0924 21:08:30.378024 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0924 21:08:30.378061 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0924 21:08:30.378108 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 21:08:30.378359 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 21:08:30.378360 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 21:08:30.378361 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 21:08:30.378389 (+ 28us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:30.378390 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:30.379655 (+ 1265us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:30.379656 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:30.380694 (+ 1038us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:30.380696 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:30.381550 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:30.381551 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:30.381610 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:30.381614 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 21:08:30.381664 (+ 50us) client_negotiation.cc:240] Negotiation successful
0924 21:08:30.381700 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":53,"threads_started":1}
I20260924 21:08:30.382421 8163 tablet_service.cc:1467] Processing CreateTablet for tablet 80876089617f4026beddcc25979bdea8 (DEFAULT_TABLE table=test-table [id=72a5f85ccdb442ad83f74f9f68a58db1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 21:08:30.382462 8303 tablet_service.cc:1467] Processing CreateTablet for tablet 80876089617f4026beddcc25979bdea8 (DEFAULT_TABLE table=test-table [id=72a5f85ccdb442ad83f74f9f68a58db1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 21:08:30.382705 8163 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 80876089617f4026beddcc25979bdea8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 21:08:30.382763 8303 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 80876089617f4026beddcc25979bdea8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 21:08:30.385334 8394 tablet_bootstrap.cc:492] T 80876089617f4026beddcc25979bdea8 P 8a234a50319146b69dfea62fbce5654b: Bootstrap starting.
I20260924 21:08:30.385665 8395 tablet_bootstrap.cc:492] T 80876089617f4026beddcc25979bdea8 P 1b3ebea12b784499ad16419ad850345a: Bootstrap starting.
I20260924 21:08:30.386059 8394 tablet_bootstrap.cc:654] T 80876089617f4026beddcc25979bdea8 P 8a234a50319146b69dfea62fbce5654b: Neither blocks nor log segments found. Creating new log.
I20260924 21:08:30.386147 8023 tablet_service.cc:1467] Processing CreateTablet for tablet 80876089617f4026beddcc25979bdea8 (DEFAULT_TABLE table=test-table [id=72a5f85ccdb442ad83f74f9f68a58db1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 21:08:30.386256 8395 tablet_bootstrap.cc:654] T 80876089617f4026beddcc25979bdea8 P 1b3ebea12b784499ad16419ad850345a: Neither blocks nor log segments found. Creating new log.
I20260924 21:08:30.386363 8394 log.cc:824] T 80876089617f4026beddcc25979bdea8 P 8a234a50319146b69dfea62fbce5654b: Log is configured to *not* fsync() on all Append() calls
I20260924 21:08:30.386445 8023 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 80876089617f4026beddcc25979bdea8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 21:08:30.386547 8395 log.cc:824] T 80876089617f4026beddcc25979bdea8 P 1b3ebea12b784499ad16419ad850345a: Log is configured to *not* fsync() on all Append() calls
I20260924 21:08:30.387022 8394 tablet_bootstrap.cc:492] T 80876089617f4026beddcc25979bdea8 P 8a234a50319146b69dfea62fbce5654b: No bootstrap required, opened a new log
I20260924 21:08:30.387069 8394 ts_tablet_manager.cc:1397] T 80876089617f4026beddcc25979bdea8 P 8a234a50319146b69dfea62fbce5654b: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 21:08:30.388254 8394 raft_consensus.cc:348] T 80876089617f4026beddcc25979bdea8 P 8a234a50319146b69dfea62fbce5654b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a234a50319146b69dfea62fbce5654b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35619 } } peers { permanent_uuid: "1b3ebea12b784499ad16419ad850345a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37145 } } peers { permanent_uuid: "efc3778361914d90b0e2c86fa65e639b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45765 } }
I20260924 21:08:30.388367 8394 raft_consensus.cc:374] T 80876089617f4026beddcc25979bdea8 P 8a234a50319146b69dfea62fbce5654b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 21:08:30.388386 8394 raft_consensus.cc:729] T 80876089617f4026beddcc25979bdea8 P 8a234a50319146b69dfea62fbce5654b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a234a50319146b69dfea62fbce5654b, State: Initialized, Role: FOLLOWER
I20260924 21:08:30.388499 8394 consensus_queue.cc:260] T 80876089617f4026beddcc25979bdea8 P 8a234a50319146b69dfea62fbce5654b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a234a50319146b69dfea62fbce5654b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35619 } } peers { permanent_uuid: "1b3ebea12b784499ad16419ad850345a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37145 } } peers { permanent_uuid: "efc3778361914d90b0e2c86fa65e639b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45765 } }
I20260924 21:08:30.388784 8394 ts_tablet_manager.cc:1428] T 80876089617f4026beddcc25979bdea8 P 8a234a50319146b69dfea62fbce5654b: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 21:08:30.388835 8369 heartbeater.cc:503] Master 127.0.98.62:42643 was elected leader, sending a full tablet report...
I20260924 21:08:30.389302 8399 tablet_bootstrap.cc:492] T 80876089617f4026beddcc25979bdea8 P efc3778361914d90b0e2c86fa65e639b: Bootstrap starting.
I20260924 21:08:30.389396 8395 tablet_bootstrap.cc:492] T 80876089617f4026beddcc25979bdea8 P 1b3ebea12b784499ad16419ad850345a: No bootstrap required, opened a new log
I20260924 21:08:30.389472 8395 ts_tablet_manager.cc:1397] T 80876089617f4026beddcc25979bdea8 P 1b3ebea12b784499ad16419ad850345a: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260924 21:08:30.390126 8399 tablet_bootstrap.cc:654] T 80876089617f4026beddcc25979bdea8 P efc3778361914d90b0e2c86fa65e639b: Neither blocks nor log segments found. Creating new log.
I20260924 21:08:30.390427 8399 log.cc:824] T 80876089617f4026beddcc25979bdea8 P efc3778361914d90b0e2c86fa65e639b: Log is configured to *not* fsync() on all Append() calls
I20260924 21:08:30.391078 8399 tablet_bootstrap.cc:492] T 80876089617f4026beddcc25979bdea8 P efc3778361914d90b0e2c86fa65e639b: No bootstrap required, opened a new log
I20260924 21:08:30.391155 8399 ts_tablet_manager.cc:1397] T 80876089617f4026beddcc25979bdea8 P efc3778361914d90b0e2c86fa65e639b: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 21:08:30.391366 8395 raft_consensus.cc:348] T 80876089617f4026beddcc25979bdea8 P 1b3ebea12b784499ad16419ad850345a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a234a50319146b69dfea62fbce5654b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35619 } } peers { permanent_uuid: "1b3ebea12b784499ad16419ad850345a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37145 } } peers { permanent_uuid: "efc3778361914d90b0e2c86fa65e639b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45765 } }
I20260924 21:08:30.391487 8395 raft_consensus.cc:374] T 80876089617f4026beddcc25979bdea8 P 1b3ebea12b784499ad16419ad850345a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 21:08:30.391508 8395 raft_consensus.cc:729] T 80876089617f4026beddcc25979bdea8 P 1b3ebea12b784499ad16419ad850345a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b3ebea12b784499ad16419ad850345a, State: Initialized, Role: FOLLOWER
I20260924 21:08:30.391628 8395 consensus_queue.cc:260] T 80876089617f4026beddcc25979bdea8 P 1b3ebea12b784499ad16419ad850345a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a234a50319146b69dfea62fbce5654b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35619 } } peers { permanent_uuid: "1b3ebea12b784499ad16419ad850345a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37145 } } peers { permanent_uuid: "efc3778361914d90b0e2c86fa65e639b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45765 } }
I20260924 21:08:30.391885 8395 ts_tablet_manager.cc:1428] T 80876089617f4026beddcc25979bdea8 P 1b3ebea12b784499ad16419ad850345a: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260924 21:08:30.391919 8229 heartbeater.cc:503] Master 127.0.98.62:42643 was elected leader, sending a full tablet report...
I20260924 21:08:30.392943 8399 raft_consensus.cc:348] T 80876089617f4026beddcc25979bdea8 P efc3778361914d90b0e2c86fa65e639b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a234a50319146b69dfea62fbce5654b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35619 } } peers { permanent_uuid: "1b3ebea12b784499ad16419ad850345a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37145 } } peers { permanent_uuid: "efc3778361914d90b0e2c86fa65e639b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45765 } }
I20260924 21:08:30.393049 8399 raft_consensus.cc:374] T 80876089617f4026beddcc25979bdea8 P efc3778361914d90b0e2c86fa65e639b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 21:08:30.393069 8399 raft_consensus.cc:729] T 80876089617f4026beddcc25979bdea8 P efc3778361914d90b0e2c86fa65e639b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: efc3778361914d90b0e2c86fa65e639b, State: Initialized, Role: FOLLOWER
I20260924 21:08:30.393162 8399 consensus_queue.cc:260] T 80876089617f4026beddcc25979bdea8 P efc3778361914d90b0e2c86fa65e639b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a234a50319146b69dfea62fbce5654b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35619 } } peers { permanent_uuid: "1b3ebea12b784499ad16419ad850345a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37145 } } peers { permanent_uuid: "efc3778361914d90b0e2c86fa65e639b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45765 } }
I20260924 21:08:30.393366 8399 ts_tablet_manager.cc:1428] T 80876089617f4026beddcc25979bdea8 P efc3778361914d90b0e2c86fa65e639b: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 21:08:30.393421 8089 heartbeater.cc:503] Master 127.0.98.62:42643 was elected leader, sending a full tablet report...
I20260924 21:08:30.399940 8401 raft_consensus.cc:482] T 80876089617f4026beddcc25979bdea8 P 1b3ebea12b784499ad16419ad850345a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 21:08:30.400040 8401 raft_consensus.cc:504] T 80876089617f4026beddcc25979bdea8 P 1b3ebea12b784499ad16419ad850345a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a234a50319146b69dfea62fbce5654b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35619 } } peers { permanent_uuid: "1b3ebea12b784499ad16419ad850345a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37145 } } peers { permanent_uuid: "efc3778361914d90b0e2c86fa65e639b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45765 } }
I20260924 21:08:30.400302 8401 leader_election.cc:288] T 80876089617f4026beddcc25979bdea8 P 1b3ebea12b784499ad16419ad850345a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a234a50319146b69dfea62fbce5654b (127.0.98.3:35619), efc3778361914d90b0e2c86fa65e639b (127.0.98.1:45765)
I20260924 21:08:30.402967 8390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:30.400501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:46471 (local address 127.0.98.3:35619)
0924 21:08:30.400538 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0924 21:08:30.400540 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 21:08:30.400547 (+ 7us) server_negotiation.cc:407] Connection header received
0924 21:08:30.400570 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:08:30.400571 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:08:30.400580 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:08:30.400643 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 21:08:30.400783 (+ 140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:30.401284 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:30.402239 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:30.402635 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:30.402698 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:30.402705 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0924 21:08:30.402873 (+ 168us) server_negotiation.cc:299] Negotiation successful
0924 21:08:30.402910 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260924 21:08:30.403015 8122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:30.400429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:35619 (local address 127.0.98.2:46471)
0924 21:08:30.400470 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0924 21:08:30.400481 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 21:08:30.400524 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 21:08:30.400655 (+ 131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 21:08:30.400656 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 21:08:30.400657 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 21:08:30.400712 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:30.400714 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:30.401352 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:30.401353 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:30.402166 (+ 813us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:30.402169 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:30.402717 (+ 548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:30.402718 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:30.402803 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:30.402806 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 21:08:30.402868 (+ 62us) client_negotiation.cc:240] Negotiation successful
0924 21:08:30.402896 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":21}
I20260924 21:08:30.403051 8393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:30.400606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:52399 (local address 127.0.98.1:45765)
0924 21:08:30.400640 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0924 21:08:30.400642 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 21:08:30.400708 (+ 66us) server_negotiation.cc:407] Connection header received
0924 21:08:30.400788 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:08:30.400789 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:08:30.400797 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:08:30.400846 (+ 49us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 21:08:30.400980 (+ 134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:30.401439 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:30.402370 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:30.402723 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:30.402791 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:30.402799 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0924 21:08:30.402954 (+ 155us) server_negotiation.cc:299] Negotiation successful
0924 21:08:30.402990 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260924 21:08:30.403286 8323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80876089617f4026beddcc25979bdea8" candidate_uuid: "1b3ebea12b784499ad16419ad850345a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a234a50319146b69dfea62fbce5654b" is_pre_election: true
I20260924 21:08:30.403373 8403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:30.400530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45765 (local address 127.0.98.2:52399)
0924 21:08:30.400665 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0924 21:08:30.400673 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0924 21:08:30.400714 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 21:08:30.400845 (+ 131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 21:08:30.400854 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 21:08:30.400855 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 21:08:30.400922 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:30.400923 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:30.401509 (+ 586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:30.401510 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:30.402289 (+ 779us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:30.402291 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:30.402797 (+ 506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:30.402798 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:30.402874 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:30.402878 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 21:08:30.402930 (+ 52us) client_negotiation.cc:240] Negotiation successful
0924 21:08:30.402964 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":78,"threads_started":1}
I20260924 21:08:30.403407 8323 raft_consensus.cc:2454] T 80876089617f4026beddcc25979bdea8 P 8a234a50319146b69dfea62fbce5654b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b3ebea12b784499ad16419ad850345a in term 0.
I20260924 21:08:30.403671 8117 leader_election.cc:302] T 80876089617f4026beddcc25979bdea8 P 1b3ebea12b784499ad16419ad850345a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b3ebea12b784499ad16419ad850345a, 8a234a50319146b69dfea62fbce5654b; no voters:
I20260924 21:08:30.403820 8401 raft_consensus.cc:2781] T 80876089617f4026beddcc25979bdea8 P 1b3ebea12b784499ad16419ad850345a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 21:08:30.403896 8401 raft_consensus.cc:482] T 80876089617f4026beddcc25979bdea8 P 1b3ebea12b784499ad16419ad850345a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 21:08:30.403921 8401 raft_consensus.cc:3037] T 80876089617f4026beddcc25979bdea8 P 1b3ebea12b784499ad16419ad850345a [term 0 FOLLOWER]: Advancing to term 1
I20260924 21:08:30.404004 8043 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80876089617f4026beddcc25979bdea8" candidate_uuid: "1b3ebea12b784499ad16419ad850345a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "efc3778361914d90b0e2c86fa65e639b" is_pre_election: true
I20260924 21:08:30.404127 8043 raft_consensus.cc:2454] T 80876089617f4026beddcc25979bdea8 P efc3778361914d90b0e2c86fa65e639b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b3ebea12b784499ad16419ad850345a in term 0.
I20260924 21:08:30.404891 8401 raft_consensus.cc:504] T 80876089617f4026beddcc25979bdea8 P 1b3ebea12b784499ad16419ad850345a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a234a50319146b69dfea62fbce5654b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35619 } } peers { permanent_uuid: "1b3ebea12b784499ad16419ad850345a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37145 } } peers { permanent_uuid: "efc3778361914d90b0e2c86fa65e639b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45765 } }
I20260924 21:08:30.404981 8401 leader_election.cc:288] T 80876089617f4026beddcc25979bdea8 P 1b3ebea12b784499ad16419ad850345a [CANDIDATE]: Term 1 election: Requested vote from peers 8a234a50319146b69dfea62fbce5654b (127.0.98.3:35619), efc3778361914d90b0e2c86fa65e639b (127.0.98.1:45765)
I20260924 21:08:30.405234 8323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80876089617f4026beddcc25979bdea8" candidate_uuid: "1b3ebea12b784499ad16419ad850345a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a234a50319146b69dfea62fbce5654b"
I20260924 21:08:30.405335 8323 raft_consensus.cc:3037] T 80876089617f4026beddcc25979bdea8 P 8a234a50319146b69dfea62fbce5654b [term 0 FOLLOWER]: Advancing to term 1
I20260924 21:08:30.406023 8043 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80876089617f4026beddcc25979bdea8" candidate_uuid: "1b3ebea12b784499ad16419ad850345a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "efc3778361914d90b0e2c86fa65e639b"
I20260924 21:08:30.406092 8043 raft_consensus.cc:3037] T 80876089617f4026beddcc25979bdea8 P efc3778361914d90b0e2c86fa65e639b [term 0 FOLLOWER]: Advancing to term 1
I20260924 21:08:30.406319 8323 raft_consensus.cc:2454] T 80876089617f4026beddcc25979bdea8 P 8a234a50319146b69dfea62fbce5654b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b3ebea12b784499ad16419ad850345a in term 1.
I20260924 21:08:30.406517 8117 leader_election.cc:302] T 80876089617f4026beddcc25979bdea8 P 1b3ebea12b784499ad16419ad850345a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b3ebea12b784499ad16419ad850345a, 8a234a50319146b69dfea62fbce5654b; no voters:
I20260924 21:08:30.406644 8401 raft_consensus.cc:2781] T 80876089617f4026beddcc25979bdea8 P 1b3ebea12b784499ad16419ad850345a [term 1 FOLLOWER]: Leader election won for term 1
I20260924 21:08:30.406760 8043 raft_consensus.cc:2454] T 80876089617f4026beddcc25979bdea8 P efc3778361914d90b0e2c86fa65e639b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b3ebea12b784499ad16419ad850345a in term 1.
I20260924 21:08:30.406781 8401 raft_consensus.cc:686] T 80876089617f4026beddcc25979bdea8 P 1b3ebea12b784499ad16419ad850345a [term 1 LEADER]: Becoming Leader. State: Replica: 1b3ebea12b784499ad16419ad850345a, State: Running, Role: LEADER
I20260924 21:08:30.406870 8401 consensus_queue.cc:237] T 80876089617f4026beddcc25979bdea8 P 1b3ebea12b784499ad16419ad850345a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a234a50319146b69dfea62fbce5654b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35619 } } peers { permanent_uuid: "1b3ebea12b784499ad16419ad850345a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37145 } } peers { permanent_uuid: "efc3778361914d90b0e2c86fa65e639b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45765 } }
I20260924 21:08:30.407569 7886 catalog_manager.cc:5495] T 80876089617f4026beddcc25979bdea8 P 1b3ebea12b784499ad16419ad850345a reported cstate change: term changed from 0 to 1, leader changed from <none> to 1b3ebea12b784499ad16419ad850345a (127.0.98.2). New cstate: current_term: 1 leader_uuid: "1b3ebea12b784499ad16419ad850345a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a234a50319146b69dfea62fbce5654b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35619 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1b3ebea12b784499ad16419ad850345a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37145 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "efc3778361914d90b0e2c86fa65e639b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45765 } health_report { overall_health: UNKNOWN } } }
I20260924 21:08:30.416468 8391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:30.414671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35540 (local address 127.0.98.2:37145)
0924 21:08:30.414732 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0924 21:08:30.414733 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 21:08:30.414740 (+ 7us) server_negotiation.cc:407] Connection header received
0924 21:08:30.414789 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:08:30.414789 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:08:30.414802 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:08:30.414855 (+ 53us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 21:08:30.414959 (+ 104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:30.415356 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:30.415937 (+ 581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:30.416084 (+ 147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:30.416126 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:30.416237 (+ 111us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 21:08:30.416306 (+ 69us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 21:08:30.416361 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0924 21:08:30.416389 (+ 28us) server_negotiation.cc:299] Negotiation successful
0924 21:08:30.416401 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260924 21:08:30.417917 8043 raft_consensus.cc:1261] T 80876089617f4026beddcc25979bdea8 P efc3778361914d90b0e2c86fa65e639b [term 1 FOLLOWER]: Refusing update from remote peer 1b3ebea12b784499ad16419ad850345a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 21:08:30.417917 8323 raft_consensus.cc:1261] T 80876089617f4026beddcc25979bdea8 P 8a234a50319146b69dfea62fbce5654b [term 1 FOLLOWER]: Refusing update from remote peer 1b3ebea12b784499ad16419ad850345a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 21:08:30.418232 8404 consensus_queue.cc:1035] T 80876089617f4026beddcc25979bdea8 P 1b3ebea12b784499ad16419ad850345a [LEADER]: Connected to new peer: Peer: permanent_uuid: "efc3778361914d90b0e2c86fa65e639b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 21:08:30.418334 8401 consensus_queue.cc:1035] T 80876089617f4026beddcc25979bdea8 P 1b3ebea12b784499ad16419ad850345a [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a234a50319146b69dfea62fbce5654b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35619 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 21:08:30.420804 8409 mvcc.cc:205] Tried to move back new op lower bound from 7333003716269281280 to 7333003716227207168. Current Snapshot: MvccSnapshot[applied={T|T < 7333003716269281280}]
I20260924 21:08:30.422751 7886 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40744:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 21:08:30.422821 7886 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40744:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 21:08:30.423350 7886 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6a3f903b8ae5424eb578b88c276a8ac9: Sending DeleteTablet for 3 replicas of tablet 80876089617f4026beddcc25979bdea8
I20260924 21:08:30.423592 8303 tablet_service.cc:1514] Processing DeleteTablet for tablet 80876089617f4026beddcc25979bdea8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 21:08:30 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:56618
I20260924 21:08:30.423624 8023 tablet_service.cc:1514] Processing DeleteTablet for tablet 80876089617f4026beddcc25979bdea8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 21:08:30 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:41618
I20260924 21:08:30.423632 8163 tablet_service.cc:1514] Processing DeleteTablet for tablet 80876089617f4026beddcc25979bdea8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 21:08:30 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:35536
I20260924 21:08:30.423766 8417 tablet_replica.cc:330] T 80876089617f4026beddcc25979bdea8 P 8a234a50319146b69dfea62fbce5654b: stopping tablet replica
I20260924 21:08:30.423844 8417 raft_consensus.cc:2229] T 80876089617f4026beddcc25979bdea8 P 8a234a50319146b69dfea62fbce5654b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 21:08:30.423878 7886 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40744:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 21:08:30.423998 7886 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 21:08:30.424141 8417 raft_consensus.cc:2258] T 80876089617f4026beddcc25979bdea8 P 8a234a50319146b69dfea62fbce5654b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 21:08:30.424606 8417 ts_tablet_manager.cc:1905] T 80876089617f4026beddcc25979bdea8 P 8a234a50319146b69dfea62fbce5654b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 21:08:30.424966 8303 tablet_service.cc:1467] Processing CreateTablet for tablet 435feab12a6247adb38da4bd306ece41 (DEFAULT_TABLE table=test-table [id=b7b10c9cea7a429d986076caeb80571f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 21:08:30.424973 8023 tablet_service.cc:1467] Processing CreateTablet for tablet 435feab12a6247adb38da4bd306ece41 (DEFAULT_TABLE table=test-table [id=b7b10c9cea7a429d986076caeb80571f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 21:08:30.425014 8163 tablet_service.cc:1467] Processing CreateTablet for tablet 435feab12a6247adb38da4bd306ece41 (DEFAULT_TABLE table=test-table [id=b7b10c9cea7a429d986076caeb80571f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 21:08:30.425055 8303 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 435feab12a6247adb38da4bd306ece41. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 21:08:30.425067 8023 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 435feab12a6247adb38da4bd306ece41. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 21:08:30.425087 8163 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 435feab12a6247adb38da4bd306ece41. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 21:08:30.426186 8399 tablet_bootstrap.cc:492] T 435feab12a6247adb38da4bd306ece41 P efc3778361914d90b0e2c86fa65e639b: Bootstrap starting.
I20260924 21:08:30.426191 8394 tablet_bootstrap.cc:492] T 435feab12a6247adb38da4bd306ece41 P 8a234a50319146b69dfea62fbce5654b: Bootstrap starting.
I20260924 21:08:30.426537 8394 tablet_bootstrap.cc:654] T 435feab12a6247adb38da4bd306ece41 P 8a234a50319146b69dfea62fbce5654b: Neither blocks nor log segments found. Creating new log.
I20260924 21:08:30.426744 8399 tablet_bootstrap.cc:654] T 435feab12a6247adb38da4bd306ece41 P efc3778361914d90b0e2c86fa65e639b: Neither blocks nor log segments found. Creating new log.
I20260924 21:08:30.426820 8395 tablet_bootstrap.cc:492] T 435feab12a6247adb38da4bd306ece41 P 1b3ebea12b784499ad16419ad850345a: Bootstrap starting.
I20260924 21:08:30.427114 8394 tablet_bootstrap.cc:492] T 435feab12a6247adb38da4bd306ece41 P 8a234a50319146b69dfea62fbce5654b: No bootstrap required, opened a new log
I20260924 21:08:30.427155 8394 ts_tablet_manager.cc:1397] T 435feab12a6247adb38da4bd306ece41 P 8a234a50319146b69dfea62fbce5654b: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260924 21:08:30.427175 8395 tablet_bootstrap.cc:654] T 435feab12a6247adb38da4bd306ece41 P 1b3ebea12b784499ad16419ad850345a: Neither blocks nor log segments found. Creating new log.
I20260924 21:08:30.427276 8394 raft_consensus.cc:348] T 435feab12a6247adb38da4bd306ece41 P 8a234a50319146b69dfea62fbce5654b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efc3778361914d90b0e2c86fa65e639b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45765 } } peers { permanent_uuid: "1b3ebea12b784499ad16419ad850345a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37145 } } peers { permanent_uuid: "8a234a50319146b69dfea62fbce5654b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35619 } }
I20260924 21:08:30.427346 8394 raft_consensus.cc:374] T 435feab12a6247adb38da4bd306ece41 P 8a234a50319146b69dfea62fbce5654b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 21:08:30.427366 8394 raft_consensus.cc:729] T 435feab12a6247adb38da4bd306ece41 P 8a234a50319146b69dfea62fbce5654b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a234a50319146b69dfea62fbce5654b, State: Initialized, Role: FOLLOWER
I20260924 21:08:30.427415 8394 consensus_queue.cc:260] T 435feab12a6247adb38da4bd306ece41 P 8a234a50319146b69dfea62fbce5654b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efc3778361914d90b0e2c86fa65e639b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45765 } } peers { permanent_uuid: "1b3ebea12b784499ad16419ad850345a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37145 } } peers { permanent_uuid: "8a234a50319146b69dfea62fbce5654b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35619 } }
I20260924 21:08:30.427498 8394 ts_tablet_manager.cc:1428] T 435feab12a6247adb38da4bd306ece41 P 8a234a50319146b69dfea62fbce5654b: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 21:08:30.427930 8395 tablet_bootstrap.cc:492] T 435feab12a6247adb38da4bd306ece41 P 1b3ebea12b784499ad16419ad850345a: No bootstrap required, opened a new log
I20260924 21:08:30.427973 8395 ts_tablet_manager.cc:1397] T 435feab12a6247adb38da4bd306ece41 P 1b3ebea12b784499ad16419ad850345a: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260924 21:08:30.428084 8395 raft_consensus.cc:348] T 435feab12a6247adb38da4bd306ece41 P 1b3ebea12b784499ad16419ad850345a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efc3778361914d90b0e2c86fa65e639b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45765 } } peers { permanent_uuid: "1b3ebea12b784499ad16419ad850345a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37145 } } peers { permanent_uuid: "8a234a50319146b69dfea62fbce5654b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35619 } }
I20260924 21:08:30.428148 8395 raft_consensus.cc:374] T 435feab12a6247adb38da4bd306ece41 P 1b3ebea12b784499ad16419ad850345a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 21:08:30.428170 8395 raft_consensus.cc:729] T 435feab12a6247adb38da4bd306ece41 P 1b3ebea12b784499ad16419ad850345a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b3ebea12b784499ad16419ad850345a, State: Initialized, Role: FOLLOWER
I20260924 21:08:30.428205 8395 consensus_queue.cc:260] T 435feab12a6247adb38da4bd306ece41 P 1b3ebea12b784499ad16419ad850345a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efc3778361914d90b0e2c86fa65e639b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45765 } } peers { permanent_uuid: "1b3ebea12b784499ad16419ad850345a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37145 } } peers { permanent_uuid: "8a234a50319146b69dfea62fbce5654b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35619 } }
I20260924 21:08:30.428274 8395 ts_tablet_manager.cc:1428] T 435feab12a6247adb38da4bd306ece41 P 1b3ebea12b784499ad16419ad850345a: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 21:08:30.428459 8417 ts_tablet_manager.cc:1918] T 80876089617f4026beddcc25979bdea8 P 8a234a50319146b69dfea62fbce5654b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 21:08:30.428606 8399 tablet_bootstrap.cc:492] T 435feab12a6247adb38da4bd306ece41 P efc3778361914d90b0e2c86fa65e639b: No bootstrap required, opened a new log
I20260924 21:08:30.428653 8399 ts_tablet_manager.cc:1397] T 435feab12a6247adb38da4bd306ece41 P efc3778361914d90b0e2c86fa65e639b: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 21:08:30.428685 8417 log.cc:1196] T 80876089617f4026beddcc25979bdea8 P 8a234a50319146b69dfea62fbce5654b: Deleting WAL directory at /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790284076800039-392-0/minicluster-data/ts-2/wal/wals/80876089617f4026beddcc25979bdea8
I20260924 21:08:30.428784 8399 raft_consensus.cc:348] T 435feab12a6247adb38da4bd306ece41 P efc3778361914d90b0e2c86fa65e639b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efc3778361914d90b0e2c86fa65e639b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45765 } } peers { permanent_uuid: "1b3ebea12b784499ad16419ad850345a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37145 } } peers { permanent_uuid: "8a234a50319146b69dfea62fbce5654b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35619 } }
I20260924 21:08:30.428841 8399 raft_consensus.cc:374] T 435feab12a6247adb38da4bd306ece41 P efc3778361914d90b0e2c86fa65e639b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 21:08:30.428870 8399 raft_consensus.cc:729] T 435feab12a6247adb38da4bd306ece41 P efc3778361914d90b0e2c86fa65e639b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: efc3778361914d90b0e2c86fa65e639b, State: Initialized, Role: FOLLOWER
I20260924 21:08:30.428911 8399 consensus_queue.cc:260] T 435feab12a6247adb38da4bd306ece41 P efc3778361914d90b0e2c86fa65e639b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efc3778361914d90b0e2c86fa65e639b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45765 } } peers { permanent_uuid: "1b3ebea12b784499ad16419ad850345a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37145 } } peers { permanent_uuid: "8a234a50319146b69dfea62fbce5654b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35619 } }
I20260924 21:08:30.428941 8417 ts_tablet_manager.cc:1939] T 80876089617f4026beddcc25979bdea8 P 8a234a50319146b69dfea62fbce5654b: Deleting consensus metadata
I20260924 21:08:30.428988 8399 ts_tablet_manager.cc:1428] T 435feab12a6247adb38da4bd306ece41 P efc3778361914d90b0e2c86fa65e639b: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 21:08:30.429248 7873 catalog_manager.cc:4841] TS 8a234a50319146b69dfea62fbce5654b (127.0.98.3:35619): tablet 80876089617f4026beddcc25979bdea8 (table test-table [id=72a5f85ccdb442ad83f74f9f68a58db1]) successfully deleted
I20260924 21:08:30.429917 8418 tablet_replica.cc:330] T 80876089617f4026beddcc25979bdea8 P efc3778361914d90b0e2c86fa65e639b: stopping tablet replica
I20260924 21:08:30.429996 8418 raft_consensus.cc:2229] T 80876089617f4026beddcc25979bdea8 P efc3778361914d90b0e2c86fa65e639b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 21:08:30.430097 8418 raft_consensus.cc:2258] T 80876089617f4026beddcc25979bdea8 P efc3778361914d90b0e2c86fa65e639b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 21:08:30.430254 8419 tablet_replica.cc:330] T 80876089617f4026beddcc25979bdea8 P 1b3ebea12b784499ad16419ad850345a: stopping tablet replica
I20260924 21:08:30.430337 8419 raft_consensus.cc:2229] T 80876089617f4026beddcc25979bdea8 P 1b3ebea12b784499ad16419ad850345a [term 1 LEADER]: Raft consensus shutting down.
I20260924 21:08:30.430456 8419 raft_consensus.cc:2258] T 80876089617f4026beddcc25979bdea8 P 1b3ebea12b784499ad16419ad850345a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 21:08:30.430512 8418 ts_tablet_manager.cc:1905] T 80876089617f4026beddcc25979bdea8 P efc3778361914d90b0e2c86fa65e639b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 21:08:30.430779 8419 ts_tablet_manager.cc:1905] T 80876089617f4026beddcc25979bdea8 P 1b3ebea12b784499ad16419ad850345a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 21:08:30.431851 8418 ts_tablet_manager.cc:1918] T 80876089617f4026beddcc25979bdea8 P efc3778361914d90b0e2c86fa65e639b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 21:08:30.431910 8418 log.cc:1196] T 80876089617f4026beddcc25979bdea8 P efc3778361914d90b0e2c86fa65e639b: Deleting WAL directory at /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790284076800039-392-0/minicluster-data/ts-0/wal/wals/80876089617f4026beddcc25979bdea8
I20260924 21:08:30.432127 8418 ts_tablet_manager.cc:1939] T 80876089617f4026beddcc25979bdea8 P efc3778361914d90b0e2c86fa65e639b: Deleting consensus metadata
I20260924 21:08:30.432137 8419 ts_tablet_manager.cc:1918] T 80876089617f4026beddcc25979bdea8 P 1b3ebea12b784499ad16419ad850345a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 21:08:30.432190 8419 log.cc:1196] T 80876089617f4026beddcc25979bdea8 P 1b3ebea12b784499ad16419ad850345a: Deleting WAL directory at /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790284076800039-392-0/minicluster-data/ts-1/wal/wals/80876089617f4026beddcc25979bdea8
I20260924 21:08:30.432402 8419 ts_tablet_manager.cc:1939] T 80876089617f4026beddcc25979bdea8 P 1b3ebea12b784499ad16419ad850345a: Deleting consensus metadata
I20260924 21:08:30.432467 7873 catalog_manager.cc:4841] TS efc3778361914d90b0e2c86fa65e639b (127.0.98.1:45765): tablet 80876089617f4026beddcc25979bdea8 (table test-table [id=72a5f85ccdb442ad83f74f9f68a58db1]) successfully deleted
I20260924 21:08:30.432792 7872 catalog_manager.cc:4841] TS 1b3ebea12b784499ad16419ad850345a (127.0.98.2:37145): tablet 80876089617f4026beddcc25979bdea8 (table test-table [id=72a5f85ccdb442ad83f74f9f68a58db1]) successfully deleted
W20260924 21:08:30.458379 8230 tablet.cc:2369] T 435feab12a6247adb38da4bd306ece41 P 1b3ebea12b784499ad16419ad850345a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 21:08:30.573710 8090 tablet.cc:2369] T 435feab12a6247adb38da4bd306ece41 P efc3778361914d90b0e2c86fa65e639b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 21:08:30.588117 8370 tablet.cc:2369] T 435feab12a6247adb38da4bd306ece41 P 8a234a50319146b69dfea62fbce5654b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 21:08:30.625021 8414 raft_consensus.cc:482] T 435feab12a6247adb38da4bd306ece41 P 1b3ebea12b784499ad16419ad850345a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 21:08:30.625114 8414 raft_consensus.cc:504] T 435feab12a6247adb38da4bd306ece41 P 1b3ebea12b784499ad16419ad850345a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efc3778361914d90b0e2c86fa65e639b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45765 } } peers { permanent_uuid: "1b3ebea12b784499ad16419ad850345a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37145 } } peers { permanent_uuid: "8a234a50319146b69dfea62fbce5654b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35619 } }
I20260924 21:08:30.625267 8414 leader_election.cc:288] T 435feab12a6247adb38da4bd306ece41 P 1b3ebea12b784499ad16419ad850345a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers efc3778361914d90b0e2c86fa65e639b (127.0.98.1:45765), 8a234a50319146b69dfea62fbce5654b (127.0.98.3:35619)
I20260924 21:08:30.625511 8043 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "435feab12a6247adb38da4bd306ece41" candidate_uuid: "1b3ebea12b784499ad16419ad850345a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "efc3778361914d90b0e2c86fa65e639b" is_pre_election: true
I20260924 21:08:30.625543 8323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "435feab12a6247adb38da4bd306ece41" candidate_uuid: "1b3ebea12b784499ad16419ad850345a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a234a50319146b69dfea62fbce5654b" is_pre_election: true
I20260924 21:08:30.625629 8323 raft_consensus.cc:2454] T 435feab12a6247adb38da4bd306ece41 P 8a234a50319146b69dfea62fbce5654b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b3ebea12b784499ad16419ad850345a in term 0.
I20260924 21:08:30.625628 8043 raft_consensus.cc:2454] T 435feab12a6247adb38da4bd306ece41 P efc3778361914d90b0e2c86fa65e639b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b3ebea12b784499ad16419ad850345a in term 0.
I20260924 21:08:30.625761 8117 leader_election.cc:302] T 435feab12a6247adb38da4bd306ece41 P 1b3ebea12b784499ad16419ad850345a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b3ebea12b784499ad16419ad850345a, 8a234a50319146b69dfea62fbce5654b; no voters:
I20260924 21:08:30.625916 8414 raft_consensus.cc:2781] T 435feab12a6247adb38da4bd306ece41 P 1b3ebea12b784499ad16419ad850345a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 21:08:30.625958 8414 raft_consensus.cc:482] T 435feab12a6247adb38da4bd306ece41 P 1b3ebea12b784499ad16419ad850345a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 21:08:30.625975 8414 raft_consensus.cc:3037] T 435feab12a6247adb38da4bd306ece41 P 1b3ebea12b784499ad16419ad850345a [term 0 FOLLOWER]: Advancing to term 1
I20260924 21:08:30.626607 8414 raft_consensus.cc:504] T 435feab12a6247adb38da4bd306ece41 P 1b3ebea12b784499ad16419ad850345a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efc3778361914d90b0e2c86fa65e639b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45765 } } peers { permanent_uuid: "1b3ebea12b784499ad16419ad850345a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37145 } } peers { permanent_uuid: "8a234a50319146b69dfea62fbce5654b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35619 } }
I20260924 21:08:30.626725 8414 leader_election.cc:288] T 435feab12a6247adb38da4bd306ece41 P 1b3ebea12b784499ad16419ad850345a [CANDIDATE]: Term 1 election: Requested vote from peers efc3778361914d90b0e2c86fa65e639b (127.0.98.1:45765), 8a234a50319146b69dfea62fbce5654b (127.0.98.3:35619)
I20260924 21:08:30.626900 8323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "435feab12a6247adb38da4bd306ece41" candidate_uuid: "1b3ebea12b784499ad16419ad850345a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a234a50319146b69dfea62fbce5654b"
I20260924 21:08:30.626925 8043 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "435feab12a6247adb38da4bd306ece41" candidate_uuid: "1b3ebea12b784499ad16419ad850345a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "efc3778361914d90b0e2c86fa65e639b"
I20260924 21:08:30.626983 8043 raft_consensus.cc:3037] T 435feab12a6247adb38da4bd306ece41 P efc3778361914d90b0e2c86fa65e639b [term 0 FOLLOWER]: Advancing to term 1
I20260924 21:08:30.626986 8323 raft_consensus.cc:3037] T 435feab12a6247adb38da4bd306ece41 P 8a234a50319146b69dfea62fbce5654b [term 0 FOLLOWER]: Advancing to term 1
I20260924 21:08:30.627539 8323 raft_consensus.cc:2454] T 435feab12a6247adb38da4bd306ece41 P 8a234a50319146b69dfea62fbce5654b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b3ebea12b784499ad16419ad850345a in term 1.
I20260924 21:08:30.627542 8043 raft_consensus.cc:2454] T 435feab12a6247adb38da4bd306ece41 P efc3778361914d90b0e2c86fa65e639b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b3ebea12b784499ad16419ad850345a in term 1.
I20260924 21:08:30.627724 8117 leader_election.cc:302] T 435feab12a6247adb38da4bd306ece41 P 1b3ebea12b784499ad16419ad850345a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b3ebea12b784499ad16419ad850345a, efc3778361914d90b0e2c86fa65e639b; no voters:
I20260924 21:08:30.627820 8414 raft_consensus.cc:2781] T 435feab12a6247adb38da4bd306ece41 P 1b3ebea12b784499ad16419ad850345a [term 1 FOLLOWER]: Leader election won for term 1
I20260924 21:08:30.627892 8414 raft_consensus.cc:686] T 435feab12a6247adb38da4bd306ece41 P 1b3ebea12b784499ad16419ad850345a [term 1 LEADER]: Becoming Leader. State: Replica: 1b3ebea12b784499ad16419ad850345a, State: Running, Role: LEADER
I20260924 21:08:30.627946 8414 consensus_queue.cc:237] T 435feab12a6247adb38da4bd306ece41 P 1b3ebea12b784499ad16419ad850345a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efc3778361914d90b0e2c86fa65e639b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45765 } } peers { permanent_uuid: "1b3ebea12b784499ad16419ad850345a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37145 } } peers { permanent_uuid: "8a234a50319146b69dfea62fbce5654b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35619 } }
I20260924 21:08:30.628468 7886 catalog_manager.cc:5495] T 435feab12a6247adb38da4bd306ece41 P 1b3ebea12b784499ad16419ad850345a reported cstate change: term changed from 0 to 1, leader changed from <none> to 1b3ebea12b784499ad16419ad850345a (127.0.98.2). New cstate: current_term: 1 leader_uuid: "1b3ebea12b784499ad16419ad850345a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efc3778361914d90b0e2c86fa65e639b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45765 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1b3ebea12b784499ad16419ad850345a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37145 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8a234a50319146b69dfea62fbce5654b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35619 } health_report { overall_health: UNKNOWN } } }
Sep 24 21:08:30 dist-test-slave-c7tk krb5kdc[7841](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284109, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260924 21:08:30.672333 8429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:30.666453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42643 (local address 127.0.0.1:40748)
0924 21:08:30.666590 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0924 21:08:30.666599 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 21:08:30.666675 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 21:08:30.666803 (+ 128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 21:08:30.666804 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 21:08:30.667072 (+ 268us) client_negotiation.cc:189] Negotiated authn=SASL
0924 21:08:30.667107 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:30.667109 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:30.667720 (+ 611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:30.667722 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:30.668169 (+ 447us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:30.668170 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:30.668510 (+ 340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:30.668511 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:30.668573 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:30.669030 (+ 457us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 21:08:30.669034 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 21:08:30.670375 (+ 1341us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 21:08:30.671551 (+ 1176us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 21:08:30.671556 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 21:08:30.671558 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 21:08:30.671721 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 21:08:30.671869 (+ 148us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 21:08:30.671870 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 21:08:30.671870 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 21:08:30.671917 (+ 47us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 21:08:30.672086 (+ 169us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 21:08:30.672091 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 21:08:30.672127 (+ 36us) client_negotiation.cc:769] Sending connection context
0924 21:08:30.672182 (+ 55us) client_negotiation.cc:240] Negotiation successful
0924 21:08:30.672253 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":87,"threads_started":1}
I20260924 21:08:30.672401 7936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:30.666489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40748 (local address 127.0.98.62:42643)
0924 21:08:30.666544 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0924 21:08:30.666546 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 21:08:30.666671 (+ 125us) server_negotiation.cc:407] Connection header received
0924 21:08:30.666741 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:08:30.666742 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:08:30.666752 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:08:30.666799 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:08:30.667186 (+ 387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:30.667657 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:30.668240 (+ 583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:30.668440 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:30.668512 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:30.670460 (+ 1948us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 21:08:30.670461 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 21:08:30.670461 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 21:08:30.670467 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 21:08:30.671492 (+ 1025us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:08:30.671776 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:08:30.671777 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:08:30.671777 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:08:30.671818 (+ 41us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:08:30.671961 (+ 143us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:08:30.671962 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:08:30.671962 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:08:30.672044 (+ 82us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 21:08:30.672086 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0924 21:08:30.672207 (+ 121us) server_negotiation.cc:299] Negotiation successful
0924 21:08:30.672289 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":32,"server-negotiator.queue_time_us":31}
I20260924 21:08:30.674140 7886 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:40748:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260924 21:08:30.674360 7886 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 21:08:30.675746 8303 tablet_service.cc:1467] Processing CreateTablet for tablet 98df98417448489d851827ea977f9a4f (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=64d26ba2ba154179888f3290e97ce597]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 21:08:30.675734 8023 tablet_service.cc:1467] Processing CreateTablet for tablet 98df98417448489d851827ea977f9a4f (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=64d26ba2ba154179888f3290e97ce597]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 21:08:30.675734 8163 tablet_service.cc:1467] Processing CreateTablet for tablet 98df98417448489d851827ea977f9a4f (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=64d26ba2ba154179888f3290e97ce597]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 21:08:30.675861 8303 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 98df98417448489d851827ea977f9a4f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 21:08:30.675863 8023 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 98df98417448489d851827ea977f9a4f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 21:08:30.675863 8163 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 98df98417448489d851827ea977f9a4f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 21:08:30.677135 8399 tablet_bootstrap.cc:492] T 98df98417448489d851827ea977f9a4f P efc3778361914d90b0e2c86fa65e639b: Bootstrap starting.
I20260924 21:08:30.677150 8394 tablet_bootstrap.cc:492] T 98df98417448489d851827ea977f9a4f P 8a234a50319146b69dfea62fbce5654b: Bootstrap starting.
I20260924 21:08:30.677471 8399 tablet_bootstrap.cc:654] T 98df98417448489d851827ea977f9a4f P efc3778361914d90b0e2c86fa65e639b: Neither blocks nor log segments found. Creating new log.
I20260924 21:08:30.677479 8394 tablet_bootstrap.cc:654] T 98df98417448489d851827ea977f9a4f P 8a234a50319146b69dfea62fbce5654b: Neither blocks nor log segments found. Creating new log.
I20260924 21:08:30.677758 8395 tablet_bootstrap.cc:492] T 98df98417448489d851827ea977f9a4f P 1b3ebea12b784499ad16419ad850345a: Bootstrap starting.
I20260924 21:08:30.678071 8394 tablet_bootstrap.cc:492] T 98df98417448489d851827ea977f9a4f P 8a234a50319146b69dfea62fbce5654b: No bootstrap required, opened a new log
I20260924 21:08:30.678071 8395 tablet_bootstrap.cc:654] T 98df98417448489d851827ea977f9a4f P 1b3ebea12b784499ad16419ad850345a: Neither blocks nor log segments found. Creating new log.
I20260924 21:08:30.678086 8399 tablet_bootstrap.cc:492] T 98df98417448489d851827ea977f9a4f P efc3778361914d90b0e2c86fa65e639b: No bootstrap required, opened a new log
I20260924 21:08:30.678121 8399 ts_tablet_manager.cc:1397] T 98df98417448489d851827ea977f9a4f P efc3778361914d90b0e2c86fa65e639b: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260924 21:08:30.678130 8394 ts_tablet_manager.cc:1397] T 98df98417448489d851827ea977f9a4f P 8a234a50319146b69dfea62fbce5654b: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260924 21:08:30.678249 8394 raft_consensus.cc:348] T 98df98417448489d851827ea977f9a4f P 8a234a50319146b69dfea62fbce5654b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b3ebea12b784499ad16419ad850345a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37145 } } peers { permanent_uuid: "8a234a50319146b69dfea62fbce5654b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35619 } } peers { permanent_uuid: "efc3778361914d90b0e2c86fa65e639b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45765 } }
I20260924 21:08:30.678254 8399 raft_consensus.cc:348] T 98df98417448489d851827ea977f9a4f P efc3778361914d90b0e2c86fa65e639b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b3ebea12b784499ad16419ad850345a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37145 } } peers { permanent_uuid: "8a234a50319146b69dfea62fbce5654b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35619 } } peers { permanent_uuid: "efc3778361914d90b0e2c86fa65e639b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45765 } }
I20260924 21:08:30.678328 8394 raft_consensus.cc:374] T 98df98417448489d851827ea977f9a4f P 8a234a50319146b69dfea62fbce5654b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 21:08:30.678328 8399 raft_consensus.cc:374] T 98df98417448489d851827ea977f9a4f P efc3778361914d90b0e2c86fa65e639b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 21:08:30.678352 8394 raft_consensus.cc:729] T 98df98417448489d851827ea977f9a4f P 8a234a50319146b69dfea62fbce5654b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a234a50319146b69dfea62fbce5654b, State: Initialized, Role: FOLLOWER
I20260924 21:08:30.678369 8399 raft_consensus.cc:729] T 98df98417448489d851827ea977f9a4f P efc3778361914d90b0e2c86fa65e639b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: efc3778361914d90b0e2c86fa65e639b, State: Initialized, Role: FOLLOWER
I20260924 21:08:30.678388 8394 consensus_queue.cc:260] T 98df98417448489d851827ea977f9a4f P 8a234a50319146b69dfea62fbce5654b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b3ebea12b784499ad16419ad850345a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37145 } } peers { permanent_uuid: "8a234a50319146b69dfea62fbce5654b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35619 } } peers { permanent_uuid: "efc3778361914d90b0e2c86fa65e639b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45765 } }
I20260924 21:08:30.678414 8399 consensus_queue.cc:260] T 98df98417448489d851827ea977f9a4f P efc3778361914d90b0e2c86fa65e639b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b3ebea12b784499ad16419ad850345a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37145 } } peers { permanent_uuid: "8a234a50319146b69dfea62fbce5654b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35619 } } peers { permanent_uuid: "efc3778361914d90b0e2c86fa65e639b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45765 } }
I20260924 21:08:30.678474 8394 ts_tablet_manager.cc:1428] T 98df98417448489d851827ea977f9a4f P 8a234a50319146b69dfea62fbce5654b: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 21:08:30.678512 8395 tablet_bootstrap.cc:492] T 98df98417448489d851827ea977f9a4f P 1b3ebea12b784499ad16419ad850345a: No bootstrap required, opened a new log
I20260924 21:08:30.678521 8399 ts_tablet_manager.cc:1428] T 98df98417448489d851827ea977f9a4f P efc3778361914d90b0e2c86fa65e639b: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260924 21:08:30.678560 8395 ts_tablet_manager.cc:1397] T 98df98417448489d851827ea977f9a4f P 1b3ebea12b784499ad16419ad850345a: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260924 21:08:30.678598 8398 tablet_replica.cc:439] T 98df98417448489d851827ea977f9a4f P 8a234a50319146b69dfea62fbce5654b: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b3ebea12b784499ad16419ad850345a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37145 } } peers { permanent_uuid: "8a234a50319146b69dfea62fbce5654b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35619 } } peers { permanent_uuid: "efc3778361914d90b0e2c86fa65e639b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45765 } } }
I20260924 21:08:30.678650 8395 raft_consensus.cc:348] T 98df98417448489d851827ea977f9a4f P 1b3ebea12b784499ad16419ad850345a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b3ebea12b784499ad16419ad850345a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37145 } } peers { permanent_uuid: "8a234a50319146b69dfea62fbce5654b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35619 } } peers { permanent_uuid: "efc3778361914d90b0e2c86fa65e639b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45765 } }
I20260924 21:08:30.678649 8402 tablet_replica.cc:439] T 98df98417448489d851827ea977f9a4f P efc3778361914d90b0e2c86fa65e639b: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b3ebea12b784499ad16419ad850345a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37145 } } peers { permanent_uuid: "8a234a50319146b69dfea62fbce5654b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35619 } } peers { permanent_uuid: "efc3778361914d90b0e2c86fa65e639b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45765 } } }
I20260924 21:08:30.678674 8398 tablet_replica.cc:442] T 98df98417448489d851827ea977f9a4f P 8a234a50319146b69dfea62fbce5654b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 21:08:30.678701 8395 raft_consensus.cc:374] T 98df98417448489d851827ea977f9a4f P 1b3ebea12b784499ad16419ad850345a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 21:08:30.678705 8402 tablet_replica.cc:442] T 98df98417448489d851827ea977f9a4f P efc3778361914d90b0e2c86fa65e639b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 21:08:30.678730 8395 raft_consensus.cc:729] T 98df98417448489d851827ea977f9a4f P 1b3ebea12b784499ad16419ad850345a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b3ebea12b784499ad16419ad850345a, State: Initialized, Role: FOLLOWER
I20260924 21:08:30.678766 8395 consensus_queue.cc:260] T 98df98417448489d851827ea977f9a4f P 1b3ebea12b784499ad16419ad850345a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b3ebea12b784499ad16419ad850345a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37145 } } peers { permanent_uuid: "8a234a50319146b69dfea62fbce5654b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35619 } } peers { permanent_uuid: "efc3778361914d90b0e2c86fa65e639b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45765 } }
I20260924 21:08:30.678853 8395 ts_tablet_manager.cc:1428] T 98df98417448489d851827ea977f9a4f P 1b3ebea12b784499ad16419ad850345a: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 21:08:30.678936 8414 tablet_replica.cc:439] T 98df98417448489d851827ea977f9a4f P 1b3ebea12b784499ad16419ad850345a: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b3ebea12b784499ad16419ad850345a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37145 } } peers { permanent_uuid: "8a234a50319146b69dfea62fbce5654b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35619 } } peers { permanent_uuid: "efc3778361914d90b0e2c86fa65e639b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45765 } } }
I20260924 21:08:30.679003 8414 tablet_replica.cc:442] T 98df98417448489d851827ea977f9a4f P 1b3ebea12b784499ad16419ad850345a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 21:08:30.811556 8402 raft_consensus.cc:482] T 98df98417448489d851827ea977f9a4f P efc3778361914d90b0e2c86fa65e639b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 21:08:30.811684 8402 raft_consensus.cc:504] T 98df98417448489d851827ea977f9a4f P efc3778361914d90b0e2c86fa65e639b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b3ebea12b784499ad16419ad850345a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37145 } } peers { permanent_uuid: "8a234a50319146b69dfea62fbce5654b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35619 } } peers { permanent_uuid: "efc3778361914d90b0e2c86fa65e639b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45765 } }
I20260924 21:08:30.811996 8402 leader_election.cc:288] T 98df98417448489d851827ea977f9a4f P efc3778361914d90b0e2c86fa65e639b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1b3ebea12b784499ad16419ad850345a (127.0.98.2:37145), 8a234a50319146b69dfea62fbce5654b (127.0.98.3:35619)
I20260924 21:08:30.814647 8391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:30.812202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:35225 (local address 127.0.98.2:37145)
0924 21:08:30.812247 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0924 21:08:30.812249 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 21:08:30.812344 (+ 95us) server_negotiation.cc:407] Connection header received
0924 21:08:30.812414 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:08:30.812415 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:08:30.812423 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:08:30.812484 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 21:08:30.812630 (+ 146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:30.813056 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:30.814026 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:30.814345 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:30.814418 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:30.814428 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0924 21:08:30.814556 (+ 128us) server_negotiation.cc:299] Negotiation successful
0924 21:08:30.814588 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260924 21:08:30.814659 8433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:30.812138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:37145 (local address 127.0.98.1:35225)
0924 21:08:30.812282 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0924 21:08:30.812290 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0924 21:08:30.812365 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 21:08:30.812488 (+ 123us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 21:08:30.812488 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 21:08:30.812490 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 21:08:30.812574 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:30.812575 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:30.813131 (+ 556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:30.813132 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:30.813908 (+ 776us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:30.813910 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:30.814418 (+ 508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:30.814419 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:30.814486 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:30.814489 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 21:08:30.814551 (+ 62us) client_negotiation.cc:240] Negotiation successful
0924 21:08:30.814585 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":82,"threads_started":1}
I20260924 21:08:30.814711 8390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:30.812227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:35739 (local address 127.0.98.3:35619)
0924 21:08:30.812263 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0924 21:08:30.812265 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 21:08:30.812377 (+ 112us) server_negotiation.cc:407] Connection header received
0924 21:08:30.812440 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:08:30.812441 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:08:30.812449 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:08:30.812492 (+ 43us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 21:08:30.812621 (+ 129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:30.813057 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:30.814077 (+ 1020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:30.814395 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:30.814475 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:30.814484 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0924 21:08:30.814615 (+ 131us) server_negotiation.cc:299] Negotiation successful
0924 21:08:30.814647 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260924 21:08:30.814939 8434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:30.812153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:35619 (local address 127.0.98.1:35739)
0924 21:08:30.812304 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0924 21:08:30.812311 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0924 21:08:30.812386 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 21:08:30.812500 (+ 114us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 21:08:30.812501 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 21:08:30.812503 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 21:08:30.812554 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:30.812556 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:30.813133 (+ 577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:30.813134 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:30.813974 (+ 840us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:30.813977 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:30.814477 (+ 500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:30.814478 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:30.814539 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:30.814542 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 21:08:30.814610 (+ 68us) client_negotiation.cc:240] Negotiation successful
0924 21:08:30.814646 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"mutex_wait_us":11,"thread_start_us":69,"threads_started":1}
I20260924 21:08:30.815011 8183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98df98417448489d851827ea977f9a4f" candidate_uuid: "efc3778361914d90b0e2c86fa65e639b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b3ebea12b784499ad16419ad850345a" is_pre_election: true
I20260924 21:08:30.815135 8183 raft_consensus.cc:2454] T 98df98417448489d851827ea977f9a4f P 1b3ebea12b784499ad16419ad850345a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate efc3778361914d90b0e2c86fa65e639b in term 0.
I20260924 21:08:30.815200 8323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98df98417448489d851827ea977f9a4f" candidate_uuid: "efc3778361914d90b0e2c86fa65e639b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a234a50319146b69dfea62fbce5654b" is_pre_election: true
I20260924 21:08:30.815281 8323 raft_consensus.cc:2454] T 98df98417448489d851827ea977f9a4f P 8a234a50319146b69dfea62fbce5654b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate efc3778361914d90b0e2c86fa65e639b in term 0.
I20260924 21:08:30.815300 7976 leader_election.cc:302] T 98df98417448489d851827ea977f9a4f P efc3778361914d90b0e2c86fa65e639b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b3ebea12b784499ad16419ad850345a, efc3778361914d90b0e2c86fa65e639b; no voters:
I20260924 21:08:30.815410 8402 raft_consensus.cc:2781] T 98df98417448489d851827ea977f9a4f P efc3778361914d90b0e2c86fa65e639b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 21:08:30.815464 8402 raft_consensus.cc:482] T 98df98417448489d851827ea977f9a4f P efc3778361914d90b0e2c86fa65e639b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 21:08:30.815491 8402 raft_consensus.cc:3037] T 98df98417448489d851827ea977f9a4f P efc3778361914d90b0e2c86fa65e639b [term 0 FOLLOWER]: Advancing to term 1
I20260924 21:08:30.816066 8402 raft_consensus.cc:504] T 98df98417448489d851827ea977f9a4f P efc3778361914d90b0e2c86fa65e639b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b3ebea12b784499ad16419ad850345a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37145 } } peers { permanent_uuid: "8a234a50319146b69dfea62fbce5654b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35619 } } peers { permanent_uuid: "efc3778361914d90b0e2c86fa65e639b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45765 } }
I20260924 21:08:30.816187 8402 leader_election.cc:288] T 98df98417448489d851827ea977f9a4f P efc3778361914d90b0e2c86fa65e639b [CANDIDATE]: Term 1 election: Requested vote from peers 1b3ebea12b784499ad16419ad850345a (127.0.98.2:37145), 8a234a50319146b69dfea62fbce5654b (127.0.98.3:35619)
I20260924 21:08:30.816309 8183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98df98417448489d851827ea977f9a4f" candidate_uuid: "efc3778361914d90b0e2c86fa65e639b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b3ebea12b784499ad16419ad850345a"
I20260924 21:08:30.816365 8183 raft_consensus.cc:3037] T 98df98417448489d851827ea977f9a4f P 1b3ebea12b784499ad16419ad850345a [term 0 FOLLOWER]: Advancing to term 1
I20260924 21:08:30.816344 8323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98df98417448489d851827ea977f9a4f" candidate_uuid: "efc3778361914d90b0e2c86fa65e639b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a234a50319146b69dfea62fbce5654b"
I20260924 21:08:30.816411 8323 raft_consensus.cc:3037] T 98df98417448489d851827ea977f9a4f P 8a234a50319146b69dfea62fbce5654b [term 0 FOLLOWER]: Advancing to term 1
I20260924 21:08:30.816903 8323 raft_consensus.cc:2454] T 98df98417448489d851827ea977f9a4f P 8a234a50319146b69dfea62fbce5654b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate efc3778361914d90b0e2c86fa65e639b in term 1.
I20260924 21:08:30.816989 8183 raft_consensus.cc:2454] T 98df98417448489d851827ea977f9a4f P 1b3ebea12b784499ad16419ad850345a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate efc3778361914d90b0e2c86fa65e639b in term 1.
I20260924 21:08:30.817081 7977 leader_election.cc:302] T 98df98417448489d851827ea977f9a4f P efc3778361914d90b0e2c86fa65e639b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8a234a50319146b69dfea62fbce5654b, efc3778361914d90b0e2c86fa65e639b; no voters:
I20260924 21:08:30.817176 8402 raft_consensus.cc:2781] T 98df98417448489d851827ea977f9a4f P efc3778361914d90b0e2c86fa65e639b [term 1 FOLLOWER]: Leader election won for term 1
I20260924 21:08:30.817284 8402 raft_consensus.cc:686] T 98df98417448489d851827ea977f9a4f P efc3778361914d90b0e2c86fa65e639b [term 1 LEADER]: Becoming Leader. State: Replica: efc3778361914d90b0e2c86fa65e639b, State: Running, Role: LEADER
I20260924 21:08:30.817376 8402 consensus_queue.cc:237] T 98df98417448489d851827ea977f9a4f P efc3778361914d90b0e2c86fa65e639b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b3ebea12b784499ad16419ad850345a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37145 } } peers { permanent_uuid: "8a234a50319146b69dfea62fbce5654b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35619 } } peers { permanent_uuid: "efc3778361914d90b0e2c86fa65e639b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45765 } }
I20260924 21:08:30.817669 8435 tablet_replica.cc:439] T 98df98417448489d851827ea977f9a4f P efc3778361914d90b0e2c86fa65e639b: TxnStatusTablet state changed. Reason: New leader efc3778361914d90b0e2c86fa65e639b. Latest consensus state: current_term: 1 leader_uuid: "efc3778361914d90b0e2c86fa65e639b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b3ebea12b784499ad16419ad850345a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37145 } } peers { permanent_uuid: "8a234a50319146b69dfea62fbce5654b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35619 } } peers { permanent_uuid: "efc3778361914d90b0e2c86fa65e639b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45765 } } }
I20260924 21:08:30.817754 8435 tablet_replica.cc:442] T 98df98417448489d851827ea977f9a4f P efc3778361914d90b0e2c86fa65e639b: This TxnStatusTablet replica's current role is: LEADER
I20260924 21:08:30.817929 8437 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260924 21:08:30.818058 8437 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260924 21:08:30.818030 7885 catalog_manager.cc:5495] T 98df98417448489d851827ea977f9a4f P efc3778361914d90b0e2c86fa65e639b reported cstate change: term changed from 0 to 1, leader changed from <none> to efc3778361914d90b0e2c86fa65e639b (127.0.98.1). New cstate: current_term: 1 leader_uuid: "efc3778361914d90b0e2c86fa65e639b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b3ebea12b784499ad16419ad850345a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37145 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8a234a50319146b69dfea62fbce5654b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35619 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "efc3778361914d90b0e2c86fa65e639b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45765 } health_report { overall_health: HEALTHY } } }
I20260924 21:08:30.859562 8393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:30.856980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41624 (local address 127.0.98.1:45765)
0924 21:08:30.857133 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0924 21:08:30.857135 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 21:08:30.857142 (+ 7us) server_negotiation.cc:407] Connection header received
0924 21:08:30.857168 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:08:30.857169 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:08:30.857183 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:08:30.857242 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 21:08:30.857385 (+ 143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:30.857886 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:30.858846 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:30.859002 (+ 156us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:30.859045 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:30.859236 (+ 191us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 21:08:30.859340 (+ 104us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 21:08:30.859398 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0924 21:08:30.859475 (+ 77us) server_negotiation.cc:299] Negotiation successful
0924 21:08:30.859492 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135}
I20260924 21:08:30.859591 8429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:30.856904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45765 (local address 127.0.0.1:41624)
0924 21:08:30.857002 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0924 21:08:30.857013 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 21:08:30.857054 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 21:08:30.857269 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 21:08:30.857270 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 21:08:30.857271 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 21:08:30.857319 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:30.857322 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:30.857953 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:30.857954 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:30.858773 (+ 819us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:30.858775 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:30.859066 (+ 291us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:30.859067 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:30.859127 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:30.859141 (+ 14us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 21:08:30.859412 (+ 271us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 21:08:30.859418 (+ 6us) client_negotiation.cc:769] Sending connection context
0924 21:08:30.859468 (+ 50us) client_negotiation.cc:240] Negotiation successful
0924 21:08:30.859512 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47}
I20260924 21:08:30.860618 8323 raft_consensus.cc:1261] T 98df98417448489d851827ea977f9a4f P 8a234a50319146b69dfea62fbce5654b [term 1 FOLLOWER]: Refusing update from remote peer efc3778361914d90b0e2c86fa65e639b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 21:08:30.860674 8183 raft_consensus.cc:1261] T 98df98417448489d851827ea977f9a4f P 1b3ebea12b784499ad16419ad850345a [term 1 FOLLOWER]: Refusing update from remote peer efc3778361914d90b0e2c86fa65e639b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 21:08:30.860832 8435 consensus_queue.cc:1035] T 98df98417448489d851827ea977f9a4f P efc3778361914d90b0e2c86fa65e639b [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a234a50319146b69dfea62fbce5654b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35619 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 21:08:30.860935 8402 consensus_queue.cc:1035] T 98df98417448489d851827ea977f9a4f P efc3778361914d90b0e2c86fa65e639b [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b3ebea12b784499ad16419ad850345a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37145 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 21:08:30.861410 8414 tablet_replica.cc:439] T 98df98417448489d851827ea977f9a4f P 1b3ebea12b784499ad16419ad850345a: TxnStatusTablet state changed. Reason: New leader efc3778361914d90b0e2c86fa65e639b. Latest consensus state: current_term: 1 leader_uuid: "efc3778361914d90b0e2c86fa65e639b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b3ebea12b784499ad16419ad850345a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37145 } } peers { permanent_uuid: "8a234a50319146b69dfea62fbce5654b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35619 } } peers { permanent_uuid: "efc3778361914d90b0e2c86fa65e639b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45765 } } }
I20260924 21:08:30.861423 8398 tablet_replica.cc:439] T 98df98417448489d851827ea977f9a4f P 8a234a50319146b69dfea62fbce5654b: TxnStatusTablet state changed. Reason: New leader efc3778361914d90b0e2c86fa65e639b. Latest consensus state: current_term: 1 leader_uuid: "efc3778361914d90b0e2c86fa65e639b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b3ebea12b784499ad16419ad850345a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37145 } } peers { permanent_uuid: "8a234a50319146b69dfea62fbce5654b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35619 } } peers { permanent_uuid: "efc3778361914d90b0e2c86fa65e639b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45765 } } }
I20260924 21:08:30.861490 8414 tablet_replica.cc:442] T 98df98417448489d851827ea977f9a4f P 1b3ebea12b784499ad16419ad850345a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 21:08:30.861492 8398 tablet_replica.cc:442] T 98df98417448489d851827ea977f9a4f P 8a234a50319146b69dfea62fbce5654b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 21:08:30.862213 8402 tablet_replica.cc:439] T 98df98417448489d851827ea977f9a4f P efc3778361914d90b0e2c86fa65e639b: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "efc3778361914d90b0e2c86fa65e639b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b3ebea12b784499ad16419ad850345a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37145 } } peers { permanent_uuid: "8a234a50319146b69dfea62fbce5654b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35619 } } peers { permanent_uuid: "efc3778361914d90b0e2c86fa65e639b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45765 } } }
I20260924 21:08:30.862298 8402 tablet_replica.cc:442] T 98df98417448489d851827ea977f9a4f P efc3778361914d90b0e2c86fa65e639b: This TxnStatusTablet replica's current role is: LEADER
I20260924 21:08:30.862366 8408 mvcc.cc:205] Tried to move back new op lower bound from 7333003718083088384 to 7333003717908365312. Current Snapshot: MvccSnapshot[applied={T|T < 7333003718083088384 or (T in {7333003718083088384})}]
I20260924 21:08:30.862682 8398 tablet_replica.cc:439] T 98df98417448489d851827ea977f9a4f P 8a234a50319146b69dfea62fbce5654b: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "efc3778361914d90b0e2c86fa65e639b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b3ebea12b784499ad16419ad850345a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37145 } } peers { permanent_uuid: "8a234a50319146b69dfea62fbce5654b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35619 } } peers { permanent_uuid: "efc3778361914d90b0e2c86fa65e639b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45765 } } }
I20260924 21:08:30.862766 8398 tablet_replica.cc:442] T 98df98417448489d851827ea977f9a4f P 8a234a50319146b69dfea62fbce5654b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 21:08:30.863389 8402 tablet_replica.cc:439] T 98df98417448489d851827ea977f9a4f P efc3778361914d90b0e2c86fa65e639b: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "efc3778361914d90b0e2c86fa65e639b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b3ebea12b784499ad16419ad850345a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37145 } } peers { permanent_uuid: "8a234a50319146b69dfea62fbce5654b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35619 } } peers { permanent_uuid: "efc3778361914d90b0e2c86fa65e639b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45765 } } }
I20260924 21:08:30.863471 8402 tablet_replica.cc:442] T 98df98417448489d851827ea977f9a4f P efc3778361914d90b0e2c86fa65e639b: This TxnStatusTablet replica's current role is: LEADER
I20260924 21:08:30.863781 8407 mvcc.cc:205] Tried to move back new op lower bound from 7333003718083088384 to 7333003717908365312. Current Snapshot: MvccSnapshot[applied={T|T < 7333003718083088384}]
I20260924 21:08:30.863759 8414 tablet_replica.cc:439] T 98df98417448489d851827ea977f9a4f P 1b3ebea12b784499ad16419ad850345a: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "efc3778361914d90b0e2c86fa65e639b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b3ebea12b784499ad16419ad850345a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37145 } } peers { permanent_uuid: "8a234a50319146b69dfea62fbce5654b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35619 } } peers { permanent_uuid: "efc3778361914d90b0e2c86fa65e639b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45765 } } }
I20260924 21:08:30.863829 8414 tablet_replica.cc:442] T 98df98417448489d851827ea977f9a4f P 1b3ebea12b784499ad16419ad850345a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 21:08:30.867929 8393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:30.865533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:55331 (local address 127.0.98.1:45765)
0924 21:08:30.865567 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0924 21:08:30.865568 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 21:08:30.865627 (+ 59us) server_negotiation.cc:407] Connection header received
0924 21:08:30.865685 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:08:30.865686 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:08:30.865694 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:08:30.865743 (+ 49us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 21:08:30.865871 (+ 128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:30.866500 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:30.867289 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:30.867585 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:30.867648 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:30.867737 (+ 89us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 21:08:30.867762 (+ 25us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 21:08:30.867801 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0924 21:08:30.867860 (+ 59us) server_negotiation.cc:299] Negotiation successful
0924 21:08:30.867874 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 21:08:30.867952 8444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:30.865483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45765 (local address 127.0.98.2:55331)
0924 21:08:30.865578 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0924 21:08:30.865586 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0924 21:08:30.865638 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 21:08:30.865745 (+ 107us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 21:08:30.865745 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 21:08:30.865746 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 21:08:30.865779 (+ 33us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:30.865780 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:30.866572 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:30.866573 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:30.867220 (+ 647us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:30.867222 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:30.867648 (+ 426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:30.867648 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:30.867681 (+ 33us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:30.867691 (+ 10us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 21:08:30.867802 (+ 111us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 21:08:30.867804 (+ 2us) client_negotiation.cc:769] Sending connection context
0924 21:08:30.867841 (+ 37us) client_negotiation.cc:240] Negotiation successful
0924 21:08:30.867898 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":67,"thread_start_us":52,"threads_started":1}
I20260924 21:08:30.869860 8043 raft_consensus.cc:1261] T 435feab12a6247adb38da4bd306ece41 P efc3778361914d90b0e2c86fa65e639b [term 1 FOLLOWER]: Refusing update from remote peer 1b3ebea12b784499ad16419ad850345a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 21:08:30.869938 8323 raft_consensus.cc:1261] T 435feab12a6247adb38da4bd306ece41 P 8a234a50319146b69dfea62fbce5654b [term 1 FOLLOWER]: Refusing update from remote peer 1b3ebea12b784499ad16419ad850345a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 21:08:30.870060 8401 consensus_queue.cc:1035] T 435feab12a6247adb38da4bd306ece41 P 1b3ebea12b784499ad16419ad850345a [LEADER]: Connected to new peer: Peer: permanent_uuid: "efc3778361914d90b0e2c86fa65e639b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 21:08:30.870126 8414 consensus_queue.cc:1035] T 435feab12a6247adb38da4bd306ece41 P 1b3ebea12b784499ad16419ad850345a [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a234a50319146b69dfea62fbce5654b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35619 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 21:08:30.880707 8391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:30.878198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:47959 (local address 127.0.98.2:37145)
0924 21:08:30.878233 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0924 21:08:30.878235 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 21:08:30.878320 (+ 85us) server_negotiation.cc:407] Connection header received
0924 21:08:30.878376 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:08:30.878377 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:08:30.878384 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:08:30.878416 (+ 32us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 21:08:30.878509 (+ 93us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:30.879094 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:30.880145 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:30.880308 (+ 163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:30.880354 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:30.880542 (+ 188us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 21:08:30.880562 (+ 20us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 21:08:30.880597 (+ 35us) server_negotiation.cc:1035] Waiting for connection context
0924 21:08:30.880656 (+ 59us) server_negotiation.cc:299] Negotiation successful
0924 21:08:30.880668 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260924 21:08:30.880838 8449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:30.878149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:37145 (local address 127.0.98.1:47959)
0924 21:08:30.878265 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0924 21:08:30.878275 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 21:08:30.878327 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 21:08:30.878429 (+ 102us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 21:08:30.878429 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 21:08:30.878430 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 21:08:30.878461 (+ 31us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:30.878462 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:30.879158 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:30.879158 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:30.880067 (+ 909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:30.880069 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:30.880386 (+ 317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:30.880386 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:30.880449 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:30.880460 (+ 11us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 21:08:30.880611 (+ 151us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 21:08:30.880612 (+ 1us) client_negotiation.cc:769] Sending connection context
0924 21:08:30.880654 (+ 42us) client_negotiation.cc:240] Negotiation successful
0924 21:08:30.880689 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":89,"thread_start_us":65,"threads_started":1}
I20260924 21:08:30.885886 7886 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40744:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 21:08:30.885967 7886 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40744:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 21:08:30.886466 7886 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6a3f903b8ae5424eb578b88c276a8ac9: Sending DeleteTablet for 3 replicas of tablet 435feab12a6247adb38da4bd306ece41
I20260924 21:08:30.886682 8163 tablet_service.cc:1514] Processing DeleteTablet for tablet 435feab12a6247adb38da4bd306ece41 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 21:08:30 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:35536
I20260924 21:08:30.886694 8023 tablet_service.cc:1514] Processing DeleteTablet for tablet 435feab12a6247adb38da4bd306ece41 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 21:08:30 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:41618
I20260924 21:08:30.886736 8303 tablet_service.cc:1514] Processing DeleteTablet for tablet 435feab12a6247adb38da4bd306ece41 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 21:08:30 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:56618
I20260924 21:08:30.886778 8419 tablet_replica.cc:330] T 435feab12a6247adb38da4bd306ece41 P 1b3ebea12b784499ad16419ad850345a: stopping tablet replica
I20260924 21:08:30.886797 8417 tablet_replica.cc:330] T 435feab12a6247adb38da4bd306ece41 P 8a234a50319146b69dfea62fbce5654b: stopping tablet replica
I20260924 21:08:30.886794 8418 tablet_replica.cc:330] T 435feab12a6247adb38da4bd306ece41 P efc3778361914d90b0e2c86fa65e639b: stopping tablet replica
I20260924 21:08:30.886821 8419 raft_consensus.cc:2229] T 435feab12a6247adb38da4bd306ece41 P 1b3ebea12b784499ad16419ad850345a [term 1 LEADER]: Raft consensus shutting down.
I20260924 21:08:30.886837 8417 raft_consensus.cc:2229] T 435feab12a6247adb38da4bd306ece41 P 8a234a50319146b69dfea62fbce5654b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 21:08:30.886843 8418 raft_consensus.cc:2229] T 435feab12a6247adb38da4bd306ece41 P efc3778361914d90b0e2c86fa65e639b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 21:08:30.886862 8417 raft_consensus.cc:2258] T 435feab12a6247adb38da4bd306ece41 P 8a234a50319146b69dfea62fbce5654b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 21:08:30.886870 8418 raft_consensus.cc:2258] T 435feab12a6247adb38da4bd306ece41 P efc3778361914d90b0e2c86fa65e639b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 21:08:30.886875 8419 raft_consensus.cc:2258] T 435feab12a6247adb38da4bd306ece41 P 1b3ebea12b784499ad16419ad850345a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 21:08:30.887112 8418 ts_tablet_manager.cc:1905] T 435feab12a6247adb38da4bd306ece41 P efc3778361914d90b0e2c86fa65e639b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 21:08:30.887116 8417 ts_tablet_manager.cc:1905] T 435feab12a6247adb38da4bd306ece41 P 8a234a50319146b69dfea62fbce5654b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 21:08:30.887133 8419 ts_tablet_manager.cc:1905] T 435feab12a6247adb38da4bd306ece41 P 1b3ebea12b784499ad16419ad850345a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 21:08:30.888660 8418 ts_tablet_manager.cc:1918] T 435feab12a6247adb38da4bd306ece41 P efc3778361914d90b0e2c86fa65e639b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 21:08:30.888716 8418 log.cc:1196] T 435feab12a6247adb38da4bd306ece41 P efc3778361914d90b0e2c86fa65e639b: Deleting WAL directory at /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790284076800039-392-0/minicluster-data/ts-0/wal/wals/435feab12a6247adb38da4bd306ece41
I20260924 21:08:30.888744 8417 ts_tablet_manager.cc:1918] T 435feab12a6247adb38da4bd306ece41 P 8a234a50319146b69dfea62fbce5654b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 21:08:30.888794 8417 log.cc:1196] T 435feab12a6247adb38da4bd306ece41 P 8a234a50319146b69dfea62fbce5654b: Deleting WAL directory at /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790284076800039-392-0/minicluster-data/ts-2/wal/wals/435feab12a6247adb38da4bd306ece41
I20260924 21:08:30.888926 8418 ts_tablet_manager.cc:1939] T 435feab12a6247adb38da4bd306ece41 P efc3778361914d90b0e2c86fa65e639b: Deleting consensus metadata
I20260924 21:08:30.889015 8417 ts_tablet_manager.cc:1939] T 435feab12a6247adb38da4bd306ece41 P 8a234a50319146b69dfea62fbce5654b: Deleting consensus metadata
I20260924 21:08:30.889132 8419 ts_tablet_manager.cc:1918] T 435feab12a6247adb38da4bd306ece41 P 1b3ebea12b784499ad16419ad850345a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 21:08:30.889179 8419 log.cc:1196] T 435feab12a6247adb38da4bd306ece41 P 1b3ebea12b784499ad16419ad850345a: Deleting WAL directory at /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790284076800039-392-0/minicluster-data/ts-1/wal/wals/435feab12a6247adb38da4bd306ece41
I20260924 21:08:30.889299 7873 catalog_manager.cc:4841] TS efc3778361914d90b0e2c86fa65e639b (127.0.98.1:45765): tablet 435feab12a6247adb38da4bd306ece41 (table test-table [id=b7b10c9cea7a429d986076caeb80571f]) successfully deleted
I20260924 21:08:30.889400 8419 ts_tablet_manager.cc:1939] T 435feab12a6247adb38da4bd306ece41 P 1b3ebea12b784499ad16419ad850345a: Deleting consensus metadata
I20260924 21:08:30.889452 7873 catalog_manager.cc:4841] TS 8a234a50319146b69dfea62fbce5654b (127.0.98.3:35619): tablet 435feab12a6247adb38da4bd306ece41 (table test-table [id=b7b10c9cea7a429d986076caeb80571f]) successfully deleted
I20260924 21:08:30.889688 7872 catalog_manager.cc:4841] TS 1b3ebea12b784499ad16419ad850345a (127.0.98.2:37145): tablet 435feab12a6247adb38da4bd306ece41 (table test-table [id=b7b10c9cea7a429d986076caeb80571f]) successfully deleted
Sep 24 21:08:30 dist-test-slave-c7tk krb5kdc[7841](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284110, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 24 21:08:30 dist-test-slave-c7tk krb5kdc[7841](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 21:08:30 dist-test-slave-c7tk krb5kdc[7841](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260924 21:08:30.911610 7936 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 21:08:30.906123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40752 (local address 127.0.98.62:42643)
0924 21:08:30.906181 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0924 21:08:30.906183 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 21:08:30.906282 (+ 99us) server_negotiation.cc:407] Connection header received
0924 21:08:30.906354 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:08:30.906354 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:08:30.906361 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:08:30.906401 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:08:30.907015 (+ 614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:30.907404 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:30.908347 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:30.908501 (+ 154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:30.908546 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:30.911562 (+ 3016us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40752: BlockingRecv error: recv got EOF from 127.0.0.1:40752 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
Sep 24 21:08:30 dist-test-slave-c7tk krb5kdc[7841](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284110, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260924 21:08:30.918380 7936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:30.912381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40756 (local address 127.0.98.62:42643)
0924 21:08:30.912420 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0924 21:08:30.912422 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 21:08:30.912477 (+ 55us) server_negotiation.cc:407] Connection header received
0924 21:08:30.912530 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:08:30.912531 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:08:30.912539 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:08:30.912582 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:08:30.912942 (+ 360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:30.913555 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:30.914222 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:30.914479 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:30.914545 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:30.916227 (+ 1682us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 21:08:30.916228 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 21:08:30.916229 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 21:08:30.916234 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 21:08:30.917452 (+ 1218us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:08:30.917721 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:08:30.917722 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:08:30.917723 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:08:30.917776 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:08:30.917977 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:08:30.917978 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:08:30.917978 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:08:30.918097 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 21:08:30.918180 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0924 21:08:30.918248 (+ 68us) server_negotiation.cc:299] Negotiation successful
0924 21:08:30.918308 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 21:08:30.919730 7886 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40756:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 21:08:30.919888 7886 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 21:08:30.921159 8023 tablet_service.cc:1467] Processing CreateTablet for tablet 732a5fef10f84ca38cdca5a16f78674f (DEFAULT_TABLE table=test-table [id=b85dd68929e543448c2805c75957390d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 21:08:30.921201 8163 tablet_service.cc:1467] Processing CreateTablet for tablet 732a5fef10f84ca38cdca5a16f78674f (DEFAULT_TABLE table=test-table [id=b85dd68929e543448c2805c75957390d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 21:08:30.921267 8023 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 732a5fef10f84ca38cdca5a16f78674f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 21:08:30.921305 8163 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 732a5fef10f84ca38cdca5a16f78674f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 21:08:30.922362 8395 tablet_bootstrap.cc:492] T 732a5fef10f84ca38cdca5a16f78674f P 1b3ebea12b784499ad16419ad850345a: Bootstrap starting.
I20260924 21:08:30.922451 8303 tablet_service.cc:1467] Processing CreateTablet for tablet 732a5fef10f84ca38cdca5a16f78674f (DEFAULT_TABLE table=test-table [id=b85dd68929e543448c2805c75957390d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 21:08:30.922575 8303 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 732a5fef10f84ca38cdca5a16f78674f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 21:08:30.922565 8399 tablet_bootstrap.cc:492] T 732a5fef10f84ca38cdca5a16f78674f P efc3778361914d90b0e2c86fa65e639b: Bootstrap starting.
I20260924 21:08:30.922704 8395 tablet_bootstrap.cc:654] T 732a5fef10f84ca38cdca5a16f78674f P 1b3ebea12b784499ad16419ad850345a: Neither blocks nor log segments found. Creating new log.
I20260924 21:08:30.922927 8399 tablet_bootstrap.cc:654] T 732a5fef10f84ca38cdca5a16f78674f P efc3778361914d90b0e2c86fa65e639b: Neither blocks nor log segments found. Creating new log.
I20260924 21:08:30.923532 8394 tablet_bootstrap.cc:492] T 732a5fef10f84ca38cdca5a16f78674f P 8a234a50319146b69dfea62fbce5654b: Bootstrap starting.
I20260924 21:08:30.923872 8394 tablet_bootstrap.cc:654] T 732a5fef10f84ca38cdca5a16f78674f P 8a234a50319146b69dfea62fbce5654b: Neither blocks nor log segments found. Creating new log.
I20260924 21:08:30.923964 8395 tablet_bootstrap.cc:492] T 732a5fef10f84ca38cdca5a16f78674f P 1b3ebea12b784499ad16419ad850345a: No bootstrap required, opened a new log
I20260924 21:08:30.924008 8395 ts_tablet_manager.cc:1397] T 732a5fef10f84ca38cdca5a16f78674f P 1b3ebea12b784499ad16419ad850345a: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260924 21:08:30.924124 8395 raft_consensus.cc:348] T 732a5fef10f84ca38cdca5a16f78674f P 1b3ebea12b784499ad16419ad850345a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efc3778361914d90b0e2c86fa65e639b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45765 } } peers { permanent_uuid: "1b3ebea12b784499ad16419ad850345a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37145 } } peers { permanent_uuid: "8a234a50319146b69dfea62fbce5654b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35619 } }
I20260924 21:08:30.924196 8395 raft_consensus.cc:374] T 732a5fef10f84ca38cdca5a16f78674f P 1b3ebea12b784499ad16419ad850345a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 21:08:30.924220 8395 raft_consensus.cc:729] T 732a5fef10f84ca38cdca5a16f78674f P 1b3ebea12b784499ad16419ad850345a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b3ebea12b784499ad16419ad850345a, State: Initialized, Role: FOLLOWER
I20260924 21:08:30.924255 8395 consensus_queue.cc:260] T 732a5fef10f84ca38cdca5a16f78674f P 1b3ebea12b784499ad16419ad850345a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efc3778361914d90b0e2c86fa65e639b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45765 } } peers { permanent_uuid: "1b3ebea12b784499ad16419ad850345a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37145 } } peers { permanent_uuid: "8a234a50319146b69dfea62fbce5654b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35619 } }
I20260924 21:08:30.924332 8399 tablet_bootstrap.cc:492] T 732a5fef10f84ca38cdca5a16f78674f P efc3778361914d90b0e2c86fa65e639b: No bootstrap required, opened a new log
I20260924 21:08:30.924341 8395 ts_tablet_manager.cc:1428] T 732a5fef10f84ca38cdca5a16f78674f P 1b3ebea12b784499ad16419ad850345a: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260924 21:08:30.924387 8399 ts_tablet_manager.cc:1397] T 732a5fef10f84ca38cdca5a16f78674f P efc3778361914d90b0e2c86fa65e639b: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 21:08:30.924522 8399 raft_consensus.cc:348] T 732a5fef10f84ca38cdca5a16f78674f P efc3778361914d90b0e2c86fa65e639b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efc3778361914d90b0e2c86fa65e639b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45765 } } peers { permanent_uuid: "1b3ebea12b784499ad16419ad850345a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37145 } } peers { permanent_uuid: "8a234a50319146b69dfea62fbce5654b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35619 } }
I20260924 21:08:30.924587 8399 raft_consensus.cc:374] T 732a5fef10f84ca38cdca5a16f78674f P efc3778361914d90b0e2c86fa65e639b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 21:08:30.924604 8399 raft_consensus.cc:729] T 732a5fef10f84ca38cdca5a16f78674f P efc3778361914d90b0e2c86fa65e639b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: efc3778361914d90b0e2c86fa65e639b, State: Initialized, Role: FOLLOWER
I20260924 21:08:30.924644 8399 consensus_queue.cc:260] T 732a5fef10f84ca38cdca5a16f78674f P efc3778361914d90b0e2c86fa65e639b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efc3778361914d90b0e2c86fa65e639b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45765 } } peers { permanent_uuid: "1b3ebea12b784499ad16419ad850345a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37145 } } peers { permanent_uuid: "8a234a50319146b69dfea62fbce5654b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35619 } }
I20260924 21:08:30.924726 8399 ts_tablet_manager.cc:1428] T 732a5fef10f84ca38cdca5a16f78674f P efc3778361914d90b0e2c86fa65e639b: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260924 21:08:30.925011 8394 tablet_bootstrap.cc:492] T 732a5fef10f84ca38cdca5a16f78674f P 8a234a50319146b69dfea62fbce5654b: No bootstrap required, opened a new log
I20260924 21:08:30.925055 8394 ts_tablet_manager.cc:1397] T 732a5fef10f84ca38cdca5a16f78674f P 8a234a50319146b69dfea62fbce5654b: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 21:08:30.925191 8394 raft_consensus.cc:348] T 732a5fef10f84ca38cdca5a16f78674f P 8a234a50319146b69dfea62fbce5654b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efc3778361914d90b0e2c86fa65e639b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45765 } } peers { permanent_uuid: "1b3ebea12b784499ad16419ad850345a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37145 } } peers { permanent_uuid: "8a234a50319146b69dfea62fbce5654b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35619 } }
I20260924 21:08:30.925256 8394 raft_consensus.cc:374] T 732a5fef10f84ca38cdca5a16f78674f P 8a234a50319146b69dfea62fbce5654b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 21:08:30.925273 8394 raft_consensus.cc:729] T 732a5fef10f84ca38cdca5a16f78674f P 8a234a50319146b69dfea62fbce5654b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a234a50319146b69dfea62fbce5654b, State: Initialized, Role: FOLLOWER
I20260924 21:08:30.925307 8394 consensus_queue.cc:260] T 732a5fef10f84ca38cdca5a16f78674f P 8a234a50319146b69dfea62fbce5654b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efc3778361914d90b0e2c86fa65e639b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45765 } } peers { permanent_uuid: "1b3ebea12b784499ad16419ad850345a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37145 } } peers { permanent_uuid: "8a234a50319146b69dfea62fbce5654b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35619 } }
I20260924 21:08:30.925388 8394 ts_tablet_manager.cc:1428] T 732a5fef10f84ca38cdca5a16f78674f P 8a234a50319146b69dfea62fbce5654b: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 21:08:31.035344 8402 raft_consensus.cc:482] T 732a5fef10f84ca38cdca5a16f78674f P efc3778361914d90b0e2c86fa65e639b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 21:08:31.035441 8402 raft_consensus.cc:504] T 732a5fef10f84ca38cdca5a16f78674f P efc3778361914d90b0e2c86fa65e639b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efc3778361914d90b0e2c86fa65e639b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45765 } } peers { permanent_uuid: "1b3ebea12b784499ad16419ad850345a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37145 } } peers { permanent_uuid: "8a234a50319146b69dfea62fbce5654b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35619 } }
I20260924 21:08:31.035631 8402 leader_election.cc:288] T 732a5fef10f84ca38cdca5a16f78674f P efc3778361914d90b0e2c86fa65e639b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1b3ebea12b784499ad16419ad850345a (127.0.98.2:37145), 8a234a50319146b69dfea62fbce5654b (127.0.98.3:35619)
I20260924 21:08:31.035833 8322 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "732a5fef10f84ca38cdca5a16f78674f" candidate_uuid: "efc3778361914d90b0e2c86fa65e639b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a234a50319146b69dfea62fbce5654b" is_pre_election: true
I20260924 21:08:31.035821 8183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "732a5fef10f84ca38cdca5a16f78674f" candidate_uuid: "efc3778361914d90b0e2c86fa65e639b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b3ebea12b784499ad16419ad850345a" is_pre_election: true
I20260924 21:08:31.035933 8322 raft_consensus.cc:2454] T 732a5fef10f84ca38cdca5a16f78674f P 8a234a50319146b69dfea62fbce5654b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate efc3778361914d90b0e2c86fa65e639b in term 0.
I20260924 21:08:31.035933 8183 raft_consensus.cc:2454] T 732a5fef10f84ca38cdca5a16f78674f P 1b3ebea12b784499ad16419ad850345a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate efc3778361914d90b0e2c86fa65e639b in term 0.
I20260924 21:08:31.036078 7976 leader_election.cc:302] T 732a5fef10f84ca38cdca5a16f78674f P efc3778361914d90b0e2c86fa65e639b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b3ebea12b784499ad16419ad850345a, efc3778361914d90b0e2c86fa65e639b; no voters:
I20260924 21:08:31.036182 8402 raft_consensus.cc:2781] T 732a5fef10f84ca38cdca5a16f78674f P efc3778361914d90b0e2c86fa65e639b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 21:08:31.036222 8402 raft_consensus.cc:482] T 732a5fef10f84ca38cdca5a16f78674f P efc3778361914d90b0e2c86fa65e639b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 21:08:31.036235 8402 raft_consensus.cc:3037] T 732a5fef10f84ca38cdca5a16f78674f P efc3778361914d90b0e2c86fa65e639b [term 0 FOLLOWER]: Advancing to term 1
I20260924 21:08:31.036855 8402 raft_consensus.cc:504] T 732a5fef10f84ca38cdca5a16f78674f P efc3778361914d90b0e2c86fa65e639b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efc3778361914d90b0e2c86fa65e639b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45765 } } peers { permanent_uuid: "1b3ebea12b784499ad16419ad850345a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37145 } } peers { permanent_uuid: "8a234a50319146b69dfea62fbce5654b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35619 } }
I20260924 21:08:31.036980 8402 leader_election.cc:288] T 732a5fef10f84ca38cdca5a16f78674f P efc3778361914d90b0e2c86fa65e639b [CANDIDATE]: Term 1 election: Requested vote from peers 1b3ebea12b784499ad16419ad850345a (127.0.98.2:37145), 8a234a50319146b69dfea62fbce5654b (127.0.98.3:35619)
I20260924 21:08:31.037087 8183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "732a5fef10f84ca38cdca5a16f78674f" candidate_uuid: "efc3778361914d90b0e2c86fa65e639b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b3ebea12b784499ad16419ad850345a"
I20260924 21:08:31.037124 8322 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "732a5fef10f84ca38cdca5a16f78674f" candidate_uuid: "efc3778361914d90b0e2c86fa65e639b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a234a50319146b69dfea62fbce5654b"
I20260924 21:08:31.037165 8183 raft_consensus.cc:3037] T 732a5fef10f84ca38cdca5a16f78674f P 1b3ebea12b784499ad16419ad850345a [term 0 FOLLOWER]: Advancing to term 1
I20260924 21:08:31.037176 8322 raft_consensus.cc:3037] T 732a5fef10f84ca38cdca5a16f78674f P 8a234a50319146b69dfea62fbce5654b [term 0 FOLLOWER]: Advancing to term 1
I20260924 21:08:31.038033 8183 raft_consensus.cc:2454] T 732a5fef10f84ca38cdca5a16f78674f P 1b3ebea12b784499ad16419ad850345a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate efc3778361914d90b0e2c86fa65e639b in term 1.
I20260924 21:08:31.038229 7976 leader_election.cc:302] T 732a5fef10f84ca38cdca5a16f78674f P efc3778361914d90b0e2c86fa65e639b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b3ebea12b784499ad16419ad850345a, efc3778361914d90b0e2c86fa65e639b; no voters:
I20260924 21:08:31.038336 8402 raft_consensus.cc:2781] T 732a5fef10f84ca38cdca5a16f78674f P efc3778361914d90b0e2c86fa65e639b [term 1 FOLLOWER]: Leader election won for term 1
I20260924 21:08:31.038394 8402 raft_consensus.cc:686] T 732a5fef10f84ca38cdca5a16f78674f P efc3778361914d90b0e2c86fa65e639b [term 1 LEADER]: Becoming Leader. State: Replica: efc3778361914d90b0e2c86fa65e639b, State: Running, Role: LEADER
I20260924 21:08:31.038427 8322 raft_consensus.cc:2454] T 732a5fef10f84ca38cdca5a16f78674f P 8a234a50319146b69dfea62fbce5654b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate efc3778361914d90b0e2c86fa65e639b in term 1.
I20260924 21:08:31.038448 8402 consensus_queue.cc:237] T 732a5fef10f84ca38cdca5a16f78674f P efc3778361914d90b0e2c86fa65e639b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efc3778361914d90b0e2c86fa65e639b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45765 } } peers { permanent_uuid: "1b3ebea12b784499ad16419ad850345a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37145 } } peers { permanent_uuid: "8a234a50319146b69dfea62fbce5654b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35619 } }
I20260924 21:08:31.038985 7885 catalog_manager.cc:5495] T 732a5fef10f84ca38cdca5a16f78674f P efc3778361914d90b0e2c86fa65e639b reported cstate change: term changed from 0 to 1, leader changed from <none> to efc3778361914d90b0e2c86fa65e639b (127.0.98.1). New cstate: current_term: 1 leader_uuid: "efc3778361914d90b0e2c86fa65e639b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efc3778361914d90b0e2c86fa65e639b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45765 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1b3ebea12b784499ad16419ad850345a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37145 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8a234a50319146b69dfea62fbce5654b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35619 } health_report { overall_health: UNKNOWN } } }
I20260924 21:08:31.071449 8393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:31.069467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41626 (local address 127.0.98.1:45765)
0924 21:08:31.069505 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0924 21:08:31.069507 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 21:08:31.069515 (+ 8us) server_negotiation.cc:407] Connection header received
0924 21:08:31.069559 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:08:31.069559 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:08:31.069568 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:08:31.069615 (+ 47us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 21:08:31.069723 (+ 108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:31.070291 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:31.070960 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:31.071129 (+ 169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:31.071175 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:31.071283 (+ 108us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 21:08:31.071307 (+ 24us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 21:08:31.071350 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0924 21:08:31.071387 (+ 37us) server_negotiation.cc:299] Negotiation successful
0924 21:08:31.071401 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260924 21:08:31.072180 8183 raft_consensus.cc:1261] T 732a5fef10f84ca38cdca5a16f78674f P 1b3ebea12b784499ad16419ad850345a [term 1 FOLLOWER]: Refusing update from remote peer efc3778361914d90b0e2c86fa65e639b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 21:08:31.072180 8322 raft_consensus.cc:1261] T 732a5fef10f84ca38cdca5a16f78674f P 8a234a50319146b69dfea62fbce5654b [term 1 FOLLOWER]: Refusing update from remote peer efc3778361914d90b0e2c86fa65e639b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 21:08:31.072381 8402 consensus_queue.cc:1035] T 732a5fef10f84ca38cdca5a16f78674f P efc3778361914d90b0e2c86fa65e639b [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b3ebea12b784499ad16419ad850345a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37145 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 21:08:31.072464 8447 consensus_queue.cc:1035] T 732a5fef10f84ca38cdca5a16f78674f P efc3778361914d90b0e2c86fa65e639b [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a234a50319146b69dfea62fbce5654b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35619 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 21:08:31.076361 7886 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40756:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 21:08:31.076436 7886 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40756:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 21:08:31.076988 7886 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6a3f903b8ae5424eb578b88c276a8ac9: Sending DeleteTablet for 3 replicas of tablet 732a5fef10f84ca38cdca5a16f78674f
I20260924 21:08:31.077222 8163 tablet_service.cc:1514] Processing DeleteTablet for tablet 732a5fef10f84ca38cdca5a16f78674f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 21:08:31 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:35536
I20260924 21:08:31.077242 8023 tablet_service.cc:1514] Processing DeleteTablet for tablet 732a5fef10f84ca38cdca5a16f78674f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 21:08:31 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:41618
I20260924 21:08:31.077301 8303 tablet_service.cc:1514] Processing DeleteTablet for tablet 732a5fef10f84ca38cdca5a16f78674f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 21:08:31 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:56618
I20260924 21:08:31.077311 8418 tablet_replica.cc:330] T 732a5fef10f84ca38cdca5a16f78674f P efc3778361914d90b0e2c86fa65e639b: stopping tablet replica
I20260924 21:08:31.077369 8418 raft_consensus.cc:2229] T 732a5fef10f84ca38cdca5a16f78674f P efc3778361914d90b0e2c86fa65e639b [term 1 LEADER]: Raft consensus shutting down.
I20260924 21:08:31.077373 8419 tablet_replica.cc:330] T 732a5fef10f84ca38cdca5a16f78674f P 1b3ebea12b784499ad16419ad850345a: stopping tablet replica
I20260924 21:08:31.077415 8419 raft_consensus.cc:2229] T 732a5fef10f84ca38cdca5a16f78674f P 1b3ebea12b784499ad16419ad850345a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 21:08:31.077451 8418 raft_consensus.cc:2258] T 732a5fef10f84ca38cdca5a16f78674f P efc3778361914d90b0e2c86fa65e639b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 21:08:31.077467 8419 raft_consensus.cc:2258] T 732a5fef10f84ca38cdca5a16f78674f P 1b3ebea12b784499ad16419ad850345a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 21:08:31.077575 8417 tablet_replica.cc:330] T 732a5fef10f84ca38cdca5a16f78674f P 8a234a50319146b69dfea62fbce5654b: stopping tablet replica
I20260924 21:08:31.077648 8417 raft_consensus.cc:2229] T 732a5fef10f84ca38cdca5a16f78674f P 8a234a50319146b69dfea62fbce5654b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 21:08:31.077704 8417 pending_rounds.cc:62] T 732a5fef10f84ca38cdca5a16f78674f P 8a234a50319146b69dfea62fbce5654b: Trying to abort 2 pending ops.
I20260924 21:08:31.077732 8417 pending_rounds.cc:68] T 732a5fef10f84ca38cdca5a16f78674f P 8a234a50319146b69dfea62fbce5654b: Aborting op as it isn't in flight: id { term: 1 index: 2 } timestamp: 7333003718950060032 op_type: WRITE_OP write_request { tablet_id: "732a5fef10f84ca38cdca5a16f78674f" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } columns { name: "val" type: INT32 is_key: false is_nullable: true immutable: false } } row_operations { rows: "<redacted>""\001\003\00090\000\0001\324\000\000" indirect_data: "<redacted>""" } external_consistency_mode: CLIENT_PROPAGATED authz_token { token_data: "\010\373\244\326\325\006\"8\n\ntest-admin\022*\n b85dd68929e543448c2805c75957390d\020\001\030\001 \001(\001" signature: "<redacted>""\313\352\001\311\245\244\245\217J\206\362pj\223\273\226\004\322\303\361X\327:\234e\3062w\262\366\201\332l\341\320\317+ \274\313\026\312QW\205\343\266.x\243\026!%&\004W\202o\276B\001vB\030" signing_key_seq_num: 0 } } request_id { client_id: "68e25f47be7e4fa2ae4087ba848893c2" seq_no: 0 first_incomplete_seq_no: 0 attempt_no: 0 }
I20260924 21:08:31.077919 8417 pending_rounds.cc:68] T 732a5fef10f84ca38cdca5a16f78674f P 8a234a50319146b69dfea62fbce5654b: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7333003718813650944 op_type: NO_OP noop_request { }
I20260924 21:08:31.077958 8417 raft_consensus.cc:2866] T 732a5fef10f84ca38cdca5a16f78674f P 8a234a50319146b69dfea62fbce5654b [term 1 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20260924 21:08:31.077982 8417 raft_consensus.cc:2258] T 732a5fef10f84ca38cdca5a16f78674f P 8a234a50319146b69dfea62fbce5654b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 21:08:31.078271 8419 ts_tablet_manager.cc:1905] T 732a5fef10f84ca38cdca5a16f78674f P 1b3ebea12b784499ad16419ad850345a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 21:08:31.078835 8418 ts_tablet_manager.cc:1905] T 732a5fef10f84ca38cdca5a16f78674f P efc3778361914d90b0e2c86fa65e639b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 21:08:31.079679 8419 ts_tablet_manager.cc:1918] T 732a5fef10f84ca38cdca5a16f78674f P 1b3ebea12b784499ad16419ad850345a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 21:08:31.079725 8419 log.cc:1196] T 732a5fef10f84ca38cdca5a16f78674f P 1b3ebea12b784499ad16419ad850345a: Deleting WAL directory at /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790284076800039-392-0/minicluster-data/ts-1/wal/wals/732a5fef10f84ca38cdca5a16f78674f
I20260924 21:08:31.079933 8419 ts_tablet_manager.cc:1939] T 732a5fef10f84ca38cdca5a16f78674f P 1b3ebea12b784499ad16419ad850345a: Deleting consensus metadata
I20260924 21:08:31.080327 8418 ts_tablet_manager.cc:1918] T 732a5fef10f84ca38cdca5a16f78674f P efc3778361914d90b0e2c86fa65e639b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 21:08:31.080402 8418 log.cc:1196] T 732a5fef10f84ca38cdca5a16f78674f P efc3778361914d90b0e2c86fa65e639b: Deleting WAL directory at /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790284076800039-392-0/minicluster-data/ts-0/wal/wals/732a5fef10f84ca38cdca5a16f78674f
I20260924 21:08:31.080425 7872 catalog_manager.cc:4841] TS 1b3ebea12b784499ad16419ad850345a (127.0.98.2:37145): tablet 732a5fef10f84ca38cdca5a16f78674f (table test-table [id=b85dd68929e543448c2805c75957390d]) successfully deleted
I20260924 21:08:31.080598 8418 ts_tablet_manager.cc:1939] T 732a5fef10f84ca38cdca5a16f78674f P efc3778361914d90b0e2c86fa65e639b: Deleting consensus metadata
I20260924 21:08:31.080744 7886 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40756:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 21:08:31.080891 7886 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 21:08:31.081140 8417 ts_tablet_manager.cc:1905] T 732a5fef10f84ca38cdca5a16f78674f P 8a234a50319146b69dfea62fbce5654b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 21:08:31.082062 7873 catalog_manager.cc:4841] TS efc3778361914d90b0e2c86fa65e639b (127.0.98.1:45765): tablet 732a5fef10f84ca38cdca5a16f78674f (table test-table [id=b85dd68929e543448c2805c75957390d]) successfully deleted
I20260924 21:08:31.082122 8417 ts_tablet_manager.cc:1918] T 732a5fef10f84ca38cdca5a16f78674f P 8a234a50319146b69dfea62fbce5654b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 21:08:31.082175 8417 log.cc:1196] T 732a5fef10f84ca38cdca5a16f78674f P 8a234a50319146b69dfea62fbce5654b: Deleting WAL directory at /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790284076800039-392-0/minicluster-data/ts-2/wal/wals/732a5fef10f84ca38cdca5a16f78674f
I20260924 21:08:31.082334 8417 ts_tablet_manager.cc:1939] T 732a5fef10f84ca38cdca5a16f78674f P 8a234a50319146b69dfea62fbce5654b: Deleting consensus metadata
I20260924 21:08:31.082804 7873 catalog_manager.cc:4841] TS 8a234a50319146b69dfea62fbce5654b (127.0.98.3:35619): tablet 732a5fef10f84ca38cdca5a16f78674f (table test-table [id=b85dd68929e543448c2805c75957390d]) successfully deleted
I20260924 21:08:31.083168 8023 tablet_service.cc:1467] Processing CreateTablet for tablet 8ce03f63b51d41dabfb9337261df1138 (DEFAULT_TABLE table=test-table [id=7bcb6992b7be46bbb2d95e0504270180]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 21:08:31.083230 8303 tablet_service.cc:1467] Processing CreateTablet for tablet 8ce03f63b51d41dabfb9337261df1138 (DEFAULT_TABLE table=test-table [id=7bcb6992b7be46bbb2d95e0504270180]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 21:08:31.083262 8023 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ce03f63b51d41dabfb9337261df1138. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 21:08:31.083251 8163 tablet_service.cc:1467] Processing CreateTablet for tablet 8ce03f63b51d41dabfb9337261df1138 (DEFAULT_TABLE table=test-table [id=7bcb6992b7be46bbb2d95e0504270180]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 21:08:31.083321 8163 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ce03f63b51d41dabfb9337261df1138. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 21:08:31.083319 8303 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ce03f63b51d41dabfb9337261df1138. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 21:08:31.084257 8399 tablet_bootstrap.cc:492] T 8ce03f63b51d41dabfb9337261df1138 P efc3778361914d90b0e2c86fa65e639b: Bootstrap starting.
I20260924 21:08:31.084396 8394 tablet_bootstrap.cc:492] T 8ce03f63b51d41dabfb9337261df1138 P 8a234a50319146b69dfea62fbce5654b: Bootstrap starting.
I20260924 21:08:31.084511 8399 tablet_bootstrap.cc:654] T 8ce03f63b51d41dabfb9337261df1138 P efc3778361914d90b0e2c86fa65e639b: Neither blocks nor log segments found. Creating new log.
I20260924 21:08:31.084746 8394 tablet_bootstrap.cc:654] T 8ce03f63b51d41dabfb9337261df1138 P 8a234a50319146b69dfea62fbce5654b: Neither blocks nor log segments found. Creating new log.
I20260924 21:08:31.085011 8399 tablet_bootstrap.cc:492] T 8ce03f63b51d41dabfb9337261df1138 P efc3778361914d90b0e2c86fa65e639b: No bootstrap required, opened a new log
I20260924 21:08:31.085044 8395 tablet_bootstrap.cc:492] T 8ce03f63b51d41dabfb9337261df1138 P 1b3ebea12b784499ad16419ad850345a: Bootstrap starting.
I20260924 21:08:31.085053 8399 ts_tablet_manager.cc:1397] T 8ce03f63b51d41dabfb9337261df1138 P efc3778361914d90b0e2c86fa65e639b: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260924 21:08:31.085139 8399 raft_consensus.cc:348] T 8ce03f63b51d41dabfb9337261df1138 P efc3778361914d90b0e2c86fa65e639b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efc3778361914d90b0e2c86fa65e639b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45765 } } peers { permanent_uuid: "1b3ebea12b784499ad16419ad850345a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37145 } } peers { permanent_uuid: "8a234a50319146b69dfea62fbce5654b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35619 } }
I20260924 21:08:31.085194 8399 raft_consensus.cc:374] T 8ce03f63b51d41dabfb9337261df1138 P efc3778361914d90b0e2c86fa65e639b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 21:08:31.085212 8399 raft_consensus.cc:729] T 8ce03f63b51d41dabfb9337261df1138 P efc3778361914d90b0e2c86fa65e639b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: efc3778361914d90b0e2c86fa65e639b, State: Initialized, Role: FOLLOWER
I20260924 21:08:31.085247 8399 consensus_queue.cc:260] T 8ce03f63b51d41dabfb9337261df1138 P efc3778361914d90b0e2c86fa65e639b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efc3778361914d90b0e2c86fa65e639b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45765 } } peers { permanent_uuid: "1b3ebea12b784499ad16419ad850345a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37145 } } peers { permanent_uuid: "8a234a50319146b69dfea62fbce5654b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35619 } }
I20260924 21:08:31.085317 8399 ts_tablet_manager.cc:1428] T 8ce03f63b51d41dabfb9337261df1138 P efc3778361914d90b0e2c86fa65e639b: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 21:08:31.085333 8395 tablet_bootstrap.cc:654] T 8ce03f63b51d41dabfb9337261df1138 P 1b3ebea12b784499ad16419ad850345a: Neither blocks nor log segments found. Creating new log.
I20260924 21:08:31.085717 8394 tablet_bootstrap.cc:492] T 8ce03f63b51d41dabfb9337261df1138 P 8a234a50319146b69dfea62fbce5654b: No bootstrap required, opened a new log
I20260924 21:08:31.085749 8395 tablet_bootstrap.cc:492] T 8ce03f63b51d41dabfb9337261df1138 P 1b3ebea12b784499ad16419ad850345a: No bootstrap required, opened a new log
I20260924 21:08:31.085760 8394 ts_tablet_manager.cc:1397] T 8ce03f63b51d41dabfb9337261df1138 P 8a234a50319146b69dfea62fbce5654b: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260924 21:08:31.085783 8395 ts_tablet_manager.cc:1397] T 8ce03f63b51d41dabfb9337261df1138 P 1b3ebea12b784499ad16419ad850345a: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260924 21:08:31.085911 8394 raft_consensus.cc:348] T 8ce03f63b51d41dabfb9337261df1138 P 8a234a50319146b69dfea62fbce5654b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efc3778361914d90b0e2c86fa65e639b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45765 } } peers { permanent_uuid: "1b3ebea12b784499ad16419ad850345a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37145 } } peers { permanent_uuid: "8a234a50319146b69dfea62fbce5654b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35619 } }
I20260924 21:08:31.085896 8395 raft_consensus.cc:348] T 8ce03f63b51d41dabfb9337261df1138 P 1b3ebea12b784499ad16419ad850345a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efc3778361914d90b0e2c86fa65e639b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45765 } } peers { permanent_uuid: "1b3ebea12b784499ad16419ad850345a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37145 } } peers { permanent_uuid: "8a234a50319146b69dfea62fbce5654b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35619 } }
I20260924 21:08:31.085960 8395 raft_consensus.cc:374] T 8ce03f63b51d41dabfb9337261df1138 P 1b3ebea12b784499ad16419ad850345a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 21:08:31.085960 8394 raft_consensus.cc:374] T 8ce03f63b51d41dabfb9337261df1138 P 8a234a50319146b69dfea62fbce5654b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 21:08:31.085979 8395 raft_consensus.cc:729] T 8ce03f63b51d41dabfb9337261df1138 P 1b3ebea12b784499ad16419ad850345a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b3ebea12b784499ad16419ad850345a, State: Initialized, Role: FOLLOWER
I20260924 21:08:31.085979 8394 raft_consensus.cc:729] T 8ce03f63b51d41dabfb9337261df1138 P 8a234a50319146b69dfea62fbce5654b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a234a50319146b69dfea62fbce5654b, State: Initialized, Role: FOLLOWER
I20260924 21:08:31.086014 8395 consensus_queue.cc:260] T 8ce03f63b51d41dabfb9337261df1138 P 1b3ebea12b784499ad16419ad850345a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efc3778361914d90b0e2c86fa65e639b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45765 } } peers { permanent_uuid: "1b3ebea12b784499ad16419ad850345a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37145 } } peers { permanent_uuid: "8a234a50319146b69dfea62fbce5654b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35619 } }
I20260924 21:08:31.086021 8394 consensus_queue.cc:260] T 8ce03f63b51d41dabfb9337261df1138 P 8a234a50319146b69dfea62fbce5654b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efc3778361914d90b0e2c86fa65e639b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45765 } } peers { permanent_uuid: "1b3ebea12b784499ad16419ad850345a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37145 } } peers { permanent_uuid: "8a234a50319146b69dfea62fbce5654b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35619 } }
I20260924 21:08:31.086086 8395 ts_tablet_manager.cc:1428] T 8ce03f63b51d41dabfb9337261df1138 P 1b3ebea12b784499ad16419ad850345a: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 21:08:31.086091 8394 ts_tablet_manager.cc:1428] T 8ce03f63b51d41dabfb9337261df1138 P 8a234a50319146b69dfea62fbce5654b: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 21:08:31.129029 8402 raft_consensus.cc:482] T 8ce03f63b51d41dabfb9337261df1138 P efc3778361914d90b0e2c86fa65e639b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 21:08:31.129101 8402 raft_consensus.cc:504] T 8ce03f63b51d41dabfb9337261df1138 P efc3778361914d90b0e2c86fa65e639b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efc3778361914d90b0e2c86fa65e639b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45765 } } peers { permanent_uuid: "1b3ebea12b784499ad16419ad850345a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37145 } } peers { permanent_uuid: "8a234a50319146b69dfea62fbce5654b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35619 } }
I20260924 21:08:31.129228 8402 leader_election.cc:288] T 8ce03f63b51d41dabfb9337261df1138 P efc3778361914d90b0e2c86fa65e639b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1b3ebea12b784499ad16419ad850345a (127.0.98.2:37145), 8a234a50319146b69dfea62fbce5654b (127.0.98.3:35619)
I20260924 21:08:31.129391 8322 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ce03f63b51d41dabfb9337261df1138" candidate_uuid: "efc3778361914d90b0e2c86fa65e639b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a234a50319146b69dfea62fbce5654b" is_pre_election: true
I20260924 21:08:31.129408 8183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ce03f63b51d41dabfb9337261df1138" candidate_uuid: "efc3778361914d90b0e2c86fa65e639b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b3ebea12b784499ad16419ad850345a" is_pre_election: true
I20260924 21:08:31.129482 8322 raft_consensus.cc:2454] T 8ce03f63b51d41dabfb9337261df1138 P 8a234a50319146b69dfea62fbce5654b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate efc3778361914d90b0e2c86fa65e639b in term 0.
I20260924 21:08:31.129482 8183 raft_consensus.cc:2454] T 8ce03f63b51d41dabfb9337261df1138 P 1b3ebea12b784499ad16419ad850345a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate efc3778361914d90b0e2c86fa65e639b in term 0.
I20260924 21:08:31.129639 7976 leader_election.cc:302] T 8ce03f63b51d41dabfb9337261df1138 P efc3778361914d90b0e2c86fa65e639b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b3ebea12b784499ad16419ad850345a, efc3778361914d90b0e2c86fa65e639b; no voters:
I20260924 21:08:31.129738 8402 raft_consensus.cc:2781] T 8ce03f63b51d41dabfb9337261df1138 P efc3778361914d90b0e2c86fa65e639b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 21:08:31.129781 8402 raft_consensus.cc:482] T 8ce03f63b51d41dabfb9337261df1138 P efc3778361914d90b0e2c86fa65e639b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 21:08:31.129824 8402 raft_consensus.cc:3037] T 8ce03f63b51d41dabfb9337261df1138 P efc3778361914d90b0e2c86fa65e639b [term 0 FOLLOWER]: Advancing to term 1
I20260924 21:08:31.130388 8402 raft_consensus.cc:504] T 8ce03f63b51d41dabfb9337261df1138 P efc3778361914d90b0e2c86fa65e639b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efc3778361914d90b0e2c86fa65e639b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45765 } } peers { permanent_uuid: "1b3ebea12b784499ad16419ad850345a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37145 } } peers { permanent_uuid: "8a234a50319146b69dfea62fbce5654b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35619 } }
I20260924 21:08:31.130532 8402 leader_election.cc:288] T 8ce03f63b51d41dabfb9337261df1138 P efc3778361914d90b0e2c86fa65e639b [CANDIDATE]: Term 1 election: Requested vote from peers 1b3ebea12b784499ad16419ad850345a (127.0.98.2:37145), 8a234a50319146b69dfea62fbce5654b (127.0.98.3:35619)
I20260924 21:08:31.130627 8183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ce03f63b51d41dabfb9337261df1138" candidate_uuid: "efc3778361914d90b0e2c86fa65e639b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b3ebea12b784499ad16419ad850345a"
I20260924 21:08:31.130690 8183 raft_consensus.cc:3037] T 8ce03f63b51d41dabfb9337261df1138 P 1b3ebea12b784499ad16419ad850345a [term 0 FOLLOWER]: Advancing to term 1
I20260924 21:08:31.130671 8322 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ce03f63b51d41dabfb9337261df1138" candidate_uuid: "efc3778361914d90b0e2c86fa65e639b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a234a50319146b69dfea62fbce5654b"
I20260924 21:08:31.130718 8322 raft_consensus.cc:3037] T 8ce03f63b51d41dabfb9337261df1138 P 8a234a50319146b69dfea62fbce5654b [term 0 FOLLOWER]: Advancing to term 1
I20260924 21:08:31.131291 8322 raft_consensus.cc:2454] T 8ce03f63b51d41dabfb9337261df1138 P 8a234a50319146b69dfea62fbce5654b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate efc3778361914d90b0e2c86fa65e639b in term 1.
I20260924 21:08:31.131289 8183 raft_consensus.cc:2454] T 8ce03f63b51d41dabfb9337261df1138 P 1b3ebea12b784499ad16419ad850345a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate efc3778361914d90b0e2c86fa65e639b in term 1.
I20260924 21:08:31.131438 7977 leader_election.cc:302] T 8ce03f63b51d41dabfb9337261df1138 P efc3778361914d90b0e2c86fa65e639b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8a234a50319146b69dfea62fbce5654b, efc3778361914d90b0e2c86fa65e639b; no voters:
I20260924 21:08:31.131542 8402 raft_consensus.cc:2781] T 8ce03f63b51d41dabfb9337261df1138 P efc3778361914d90b0e2c86fa65e639b [term 1 FOLLOWER]: Leader election won for term 1
I20260924 21:08:31.131585 8402 raft_consensus.cc:686] T 8ce03f63b51d41dabfb9337261df1138 P efc3778361914d90b0e2c86fa65e639b [term 1 LEADER]: Becoming Leader. State: Replica: efc3778361914d90b0e2c86fa65e639b, State: Running, Role: LEADER
I20260924 21:08:31.131626 8402 consensus_queue.cc:237] T 8ce03f63b51d41dabfb9337261df1138 P efc3778361914d90b0e2c86fa65e639b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efc3778361914d90b0e2c86fa65e639b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45765 } } peers { permanent_uuid: "1b3ebea12b784499ad16419ad850345a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37145 } } peers { permanent_uuid: "8a234a50319146b69dfea62fbce5654b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35619 } }
I20260924 21:08:31.132077 7885 catalog_manager.cc:5495] T 8ce03f63b51d41dabfb9337261df1138 P efc3778361914d90b0e2c86fa65e639b reported cstate change: term changed from 0 to 1, leader changed from <none> to efc3778361914d90b0e2c86fa65e639b (127.0.98.1). New cstate: current_term: 1 leader_uuid: "efc3778361914d90b0e2c86fa65e639b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efc3778361914d90b0e2c86fa65e639b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45765 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1b3ebea12b784499ad16419ad850345a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37145 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8a234a50319146b69dfea62fbce5654b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35619 } health_report { overall_health: UNKNOWN } } }
I20260924 21:08:31.147941 8393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:31.145949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:39413 (local address 127.0.98.1:45765)
0924 21:08:31.145979 (+ 30us) server_negotiation.cc:206] Beginning negotiation
0924 21:08:31.145981 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 21:08:31.145989 (+ 8us) server_negotiation.cc:407] Connection header received
0924 21:08:31.146039 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:08:31.146040 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:08:31.146046 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:08:31.146084 (+ 38us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 21:08:31.146177 (+ 93us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:31.146554 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:31.147458 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:31.147613 (+ 155us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:31.147654 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:31.147777 (+ 123us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 21:08:31.147798 (+ 21us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 21:08:31.147828 (+ 30us) server_negotiation.cc:1035] Waiting for connection context
0924 21:08:31.147872 (+ 44us) server_negotiation.cc:299] Negotiation successful
0924 21:08:31.147887 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":14}
I20260924 21:08:31.148005 8449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:31.145906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45765 (local address 127.0.98.1:39413)
0924 21:08:31.145950 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0924 21:08:31.145959 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 21:08:31.145992 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 21:08:31.146088 (+ 96us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 21:08:31.146088 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 21:08:31.146089 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 21:08:31.146127 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:31.146128 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:31.146622 (+ 494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:31.146623 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:31.147384 (+ 761us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:31.147386 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:31.147671 (+ 285us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:31.147672 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:31.147717 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:31.147722 (+ 5us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 21:08:31.147829 (+ 107us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 21:08:31.147830 (+ 1us) client_negotiation.cc:769] Sending connection context
0924 21:08:31.147871 (+ 41us) client_negotiation.cc:240] Negotiation successful
0924 21:08:31.147908 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28,"mutex_wait_us":1}
I20260924 21:08:31.149436 8183 raft_consensus.cc:1261] T 8ce03f63b51d41dabfb9337261df1138 P 1b3ebea12b784499ad16419ad850345a [term 1 FOLLOWER]: Refusing update from remote peer efc3778361914d90b0e2c86fa65e639b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 21:08:31.149623 8322 raft_consensus.cc:1261] T 8ce03f63b51d41dabfb9337261df1138 P 8a234a50319146b69dfea62fbce5654b [term 1 FOLLOWER]: Refusing update from remote peer efc3778361914d90b0e2c86fa65e639b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 21:08:31.149628 8447 consensus_queue.cc:1035] T 8ce03f63b51d41dabfb9337261df1138 P efc3778361914d90b0e2c86fa65e639b [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b3ebea12b784499ad16419ad850345a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37145 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 21:08:31.149784 8447 consensus_queue.cc:1035] T 8ce03f63b51d41dabfb9337261df1138 P efc3778361914d90b0e2c86fa65e639b [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a234a50319146b69dfea62fbce5654b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35619 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 21:08:31.158740 7884 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40756:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 21:08:31.158808 7884 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40756:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 21:08:31.159379 7884 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6a3f903b8ae5424eb578b88c276a8ac9: Sending DeleteTablet for 3 replicas of tablet 8ce03f63b51d41dabfb9337261df1138
I20260924 21:08:31.159590 8023 tablet_service.cc:1514] Processing DeleteTablet for tablet 8ce03f63b51d41dabfb9337261df1138 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 21:08:31 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:41618
I20260924 21:08:31.159677 8418 tablet_replica.cc:330] T 8ce03f63b51d41dabfb9337261df1138 P efc3778361914d90b0e2c86fa65e639b: stopping tablet replica
I20260924 21:08:31.159716 8418 raft_consensus.cc:2229] T 8ce03f63b51d41dabfb9337261df1138 P efc3778361914d90b0e2c86fa65e639b [term 1 LEADER]: Raft consensus shutting down.
I20260924 21:08:31.159761 8418 raft_consensus.cc:2258] T 8ce03f63b51d41dabfb9337261df1138 P efc3778361914d90b0e2c86fa65e639b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 21:08:31.159742 8303 tablet_service.cc:1514] Processing DeleteTablet for tablet 8ce03f63b51d41dabfb9337261df1138 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 21:08:31 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:56618
I20260924 21:08:31.159746 8163 tablet_service.cc:1514] Processing DeleteTablet for tablet 8ce03f63b51d41dabfb9337261df1138 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 21:08:31 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:35536
I20260924 21:08:31.159819 8419 tablet_replica.cc:330] T 8ce03f63b51d41dabfb9337261df1138 P 1b3ebea12b784499ad16419ad850345a: stopping tablet replica
I20260924 21:08:31.159866 8419 raft_consensus.cc:2229] T 8ce03f63b51d41dabfb9337261df1138 P 1b3ebea12b784499ad16419ad850345a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 21:08:31.159920 8419 raft_consensus.cc:2258] T 8ce03f63b51d41dabfb9337261df1138 P 1b3ebea12b784499ad16419ad850345a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 21:08:31.159979 8417 tablet_replica.cc:330] T 8ce03f63b51d41dabfb9337261df1138 P 8a234a50319146b69dfea62fbce5654b: stopping tablet replica
I20260924 21:08:31.160017 8417 raft_consensus.cc:2229] T 8ce03f63b51d41dabfb9337261df1138 P 8a234a50319146b69dfea62fbce5654b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 21:08:31.160051 8417 raft_consensus.cc:2258] T 8ce03f63b51d41dabfb9337261df1138 P 8a234a50319146b69dfea62fbce5654b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 21:08:31.160048 8418 ts_tablet_manager.cc:1905] T 8ce03f63b51d41dabfb9337261df1138 P efc3778361914d90b0e2c86fa65e639b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 21:08:31.160266 8417 ts_tablet_manager.cc:1905] T 8ce03f63b51d41dabfb9337261df1138 P 8a234a50319146b69dfea62fbce5654b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 21:08:31.161466 8418 ts_tablet_manager.cc:1918] T 8ce03f63b51d41dabfb9337261df1138 P efc3778361914d90b0e2c86fa65e639b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 21:08:31.161517 8418 log.cc:1196] T 8ce03f63b51d41dabfb9337261df1138 P efc3778361914d90b0e2c86fa65e639b: Deleting WAL directory at /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790284076800039-392-0/minicluster-data/ts-0/wal/wals/8ce03f63b51d41dabfb9337261df1138
I20260924 21:08:31.161598 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAzNNjS/build/release/bin/kudu with pid 7958
I20260924 21:08:31.161651 8417 ts_tablet_manager.cc:1918] T 8ce03f63b51d41dabfb9337261df1138 P 8a234a50319146b69dfea62fbce5654b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 21:08:31.161698 8417 log.cc:1196] T 8ce03f63b51d41dabfb9337261df1138 P 8a234a50319146b69dfea62fbce5654b: Deleting WAL directory at /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790284076800039-392-0/minicluster-data/ts-2/wal/wals/8ce03f63b51d41dabfb9337261df1138
I20260924 21:08:31.161701 8418 ts_tablet_manager.cc:1939] T 8ce03f63b51d41dabfb9337261df1138 P efc3778361914d90b0e2c86fa65e639b: Deleting consensus metadata
I20260924 21:08:31.161911 8417 ts_tablet_manager.cc:1939] T 8ce03f63b51d41dabfb9337261df1138 P 8a234a50319146b69dfea62fbce5654b: Deleting consensus metadata
I20260924 21:08:31.162266 7873 catalog_manager.cc:4841] TS 8a234a50319146b69dfea62fbce5654b (127.0.98.3:35619): tablet 8ce03f63b51d41dabfb9337261df1138 (table test-table [id=7bcb6992b7be46bbb2d95e0504270180]) successfully deleted
I20260924 21:08:31.163359 8419 ts_tablet_manager.cc:1905] T 8ce03f63b51d41dabfb9337261df1138 P 1b3ebea12b784499ad16419ad850345a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 21:08:31.164371 8419 ts_tablet_manager.cc:1918] T 8ce03f63b51d41dabfb9337261df1138 P 1b3ebea12b784499ad16419ad850345a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 21:08:31.164433 8419 log.cc:1196] T 8ce03f63b51d41dabfb9337261df1138 P 1b3ebea12b784499ad16419ad850345a: Deleting WAL directory at /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790284076800039-392-0/minicluster-data/ts-1/wal/wals/8ce03f63b51d41dabfb9337261df1138
I20260924 21:08:31.164618 8419 ts_tablet_manager.cc:1939] T 8ce03f63b51d41dabfb9337261df1138 P 1b3ebea12b784499ad16419ad850345a: Deleting consensus metadata
I20260924 21:08:31.164896 7872 catalog_manager.cc:4841] TS 1b3ebea12b784499ad16419ad850345a (127.0.98.2:37145): tablet 8ce03f63b51d41dabfb9337261df1138 (table test-table [id=7bcb6992b7be46bbb2d95e0504270180]) successfully deleted
W20260924 21:08:31.167646 7873 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:45765 (error 108)
I20260924 21:08:31.167814 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAzNNjS/build/release/bin/kudu with pid 8098
I20260924 21:08:31.168550 8479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:31.168405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45765 (local address 127.0.0.1:41642)
0924 21:08:31.168484 (+ 79us) negotiation.cc:107] Waiting for socket to connect
0924 21:08:31.168518 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:45765: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65,"thread_start_us":48,"threads_started":1}
W20260924 21:08:31.168699 7873 catalog_manager.cc:4584] TS efc3778361914d90b0e2c86fa65e639b (127.0.98.1:45765): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 8ce03f63b51d41dabfb9337261df1138: Network error: Client connection negotiation failed: client connection to 127.0.98.1:45765: connect: Connection refused (error 111)
I20260924 21:08:31.173013 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAzNNjS/build/release/bin/kudu with pid 8238
I20260924 21:08:31.177210 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAzNNjS/build/release/bin/kudu with pid 7854
2026-09-24T21:08:31Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (2402 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790284076800039-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 21:08:31 dist-test-slave-c7tk krb5kdc[8482](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 21:08:31 dist-test-slave-c7tk krb5kdc[8482](info): set up 2 sockets
Sep 24 21:08:31 dist-test-slave-c7tk krb5kdc[8482](info): commencing operation
krb5kdc: starting...
W20260924 21:08:32.102285 392 mini_kdc.cc:121] Time spent starting KDC: real 0.905s user 0.002s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790284076800039-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790284076800039-392-0/krb5kdc/test-user.keytab.
Sep 24 21:08:32 dist-test-slave-c7tk krb5kdc[8482](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284112, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T21:08:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T21:08:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790284076800039-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790284076800039-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 21:08:32.204797 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAzNNjS/build/release/bin/kudu
/tmp/dist-test-taskAzNNjS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790284076800039-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790284076800039-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790284076800039-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790284076800039-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39693
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790284076800039-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43247
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39693
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790284076800039-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790284076800039-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790284076800039-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 21:08:32.274096 8495 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 21:08:32.274281 8495 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 21:08:32.274312 8495 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 21:08:32.275314 8495 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 21:08:32.275374 8495 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 21:08:32.275394 8495 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 21:08:32.275411 8495 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 21:08:32.275437 8495 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 21:08:32.276695 8495 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43247
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790284076800039-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790284076800039-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39693
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790284076800039-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39693
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790284076800039-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8495
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790284076800039-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 19:49:06 UTC on ccycloud.quasar-vhqjoc.root.comops.site
build id 4670864
I20260924 21:08:32.276921 8495 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 21:08:32.277105 8495 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 21:08:32.281309 8503 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 21:08:32.281464 8495 server_base.cc:1034] running on GCE node
W20260924 21:08:32.281309 8500 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 21:08:32.281287 8501 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 21:08:32.281769 8495 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 21:08:32.282099 8495 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 21:08:32.283226 8495 hybrid_clock.cc:648] HybridClock initialized: now 1790284112283218 us; error 25 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-24T21:08:32Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1114 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790284076800039-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 21:08:32 dist-test-slave-c7tk krb5kdc[8508](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 21:08:32 dist-test-slave-c7tk krb5kdc[8508](info): set up 2 sockets
Sep 24 21:08:32 dist-test-slave-c7tk krb5kdc[8508](info): commencing operation
krb5kdc: starting...
W20260924 21:08:33.222141 392 mini_kdc.cc:121] Time spent starting KDC: real 0.911s user 0.003s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790284076800039-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790284076800039-392-0/krb5kdc/test-user.keytab.
Sep 24 21:08:33 dist-test-slave-c7tk krb5kdc[8508](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284113, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T21:08:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T21:08:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790284076800039-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790284076800039-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 21:08:33.320245 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAzNNjS/build/release/bin/kudu
/tmp/dist-test-taskAzNNjS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790284076800039-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790284076800039-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790284076800039-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790284076800039-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34181
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790284076800039-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41267
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34181
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790284076800039-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790284076800039-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790284076800039-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 21:08:33.392482 8521 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 21:08:33.392654 8521 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 21:08:33.392674 8521 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 21:08:33.393644 8521 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 21:08:33.393682 8521 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 21:08:33.393697 8521 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 21:08:33.393710 8521 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 21:08:33.393723 8521 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 21:08:33.395051 8521 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41267
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790284076800039-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790284076800039-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34181
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790284076800039-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34181
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790284076800039-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8521
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790284076800039-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 19:49:06 UTC on ccycloud.quasar-vhqjoc.root.comops.site
build id 4670864
I20260924 21:08:33.395215 8521 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 21:08:33.395367 8521 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 21:08:33.399647 8526 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 21:08:33.399781 8529 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 21:08:33.399788 8527 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 21:08:33.399690 8521 server_base.cc:1034] running on GCE node
I20260924 21:08:33.400086 8521 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 21:08:33.400384 8521 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 21:08:33.401640 8521 hybrid_clock.cc:648] HybridClock initialized: now 1790284113401623 us; error 32 us; skew 500 ppm
Sep 24 21:08:33 dist-test-slave-c7tk krb5kdc[8508](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284113, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 21:08:33.403733 8521 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 21:08:33.404122 8521 webserver.cc:460] Webserver started at http://127.0.98.62:41211/ using document root <none> and password file <none>
I20260924 21:08:33.404311 8521 fs_manager.cc:359] Metadata directory not provided
I20260924 21:08:33.404351 8521 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 21:08:33.404497 8521 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 21:08:33.405354 8521 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790284076800039-392-0/minicluster-data/master-0/data/instance:
uuid: "e592d546539745c395ef1674f85bf480"
format_stamp: "Formatted at 2026-09-24 21:08:33 on dist-test-slave-c7tk"
server_key: "bb37e1f0c442500763bc536742e13aa3"
server_key_iv: "5e21e257844f93a6e2c18893f16e04af"
server_key_version: "encryptionkey@0"
I20260924 21:08:33.405696 8521 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790284076800039-392-0/minicluster-data/master-0/wal/instance:
uuid: "e592d546539745c395ef1674f85bf480"
format_stamp: "Formatted at 2026-09-24 21:08:33 on dist-test-slave-c7tk"
server_key: "bb37e1f0c442500763bc536742e13aa3"
server_key_iv: "5e21e257844f93a6e2c18893f16e04af"
server_key_version: "encryptionkey@0"
I20260924 21:08:33.406865 8521 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 21:08:33.407441 8536 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 21:08:33.407578 8521 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 21:08:33.407645 8521 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790284076800039-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790284076800039-392-0/minicluster-data/master-0/wal
uuid: "e592d546539745c395ef1674f85bf480"
format_stamp: "Formatted at 2026-09-24 21:08:33 on dist-test-slave-c7tk"
server_key: "bb37e1f0c442500763bc536742e13aa3"
server_key_iv: "5e21e257844f93a6e2c18893f16e04af"
server_key_version: "encryptionkey@0"
I20260924 21:08:33.407689 8521 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790284076800039-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790284076800039-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790284076800039-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 21:08:33.420977 8521 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 21:08:33.422526 8521 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 21:08:33.422616 8521 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 21:08:33.425902 8521 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34181
I20260924 21:08:33.425963 8588 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34181 every 8 connection(s)
I20260924 21:08:33.426323 8521 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790284076800039-392-0/minicluster-data/master-0/data/info.pb
I20260924 21:08:33.426834 8589 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 21:08:33.428725 8589 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e592d546539745c395ef1674f85bf480: Bootstrap starting.
I20260924 21:08:33.429235 8589 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e592d546539745c395ef1674f85bf480: Neither blocks nor log segments found. Creating new log.
I20260924 21:08:33.429522 8589 log.cc:824] T 00000000000000000000000000000000 P e592d546539745c395ef1674f85bf480: Log is configured to *not* fsync() on all Append() calls
I20260924 21:08:33.430192 8589 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e592d546539745c395ef1674f85bf480: No bootstrap required, opened a new log
I20260924 21:08:33.431550 8589 raft_consensus.cc:348] T 00000000000000000000000000000000 P e592d546539745c395ef1674f85bf480 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e592d546539745c395ef1674f85bf480" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34181 } }
I20260924 21:08:33.431656 8589 raft_consensus.cc:374] T 00000000000000000000000000000000 P e592d546539745c395ef1674f85bf480 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 21:08:33.431706 8589 raft_consensus.cc:729] T 00000000000000000000000000000000 P e592d546539745c395ef1674f85bf480 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e592d546539745c395ef1674f85bf480, State: Initialized, Role: FOLLOWER
I20260924 21:08:33.431845 8589 consensus_queue.cc:260] T 00000000000000000000000000000000 P e592d546539745c395ef1674f85bf480 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e592d546539745c395ef1674f85bf480" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34181 } }
I20260924 21:08:33.431918 8589 raft_consensus.cc:388] T 00000000000000000000000000000000 P e592d546539745c395ef1674f85bf480 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 21:08:33.431962 8589 raft_consensus.cc:482] T 00000000000000000000000000000000 P e592d546539745c395ef1674f85bf480 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 21:08:33.432003 8589 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e592d546539745c395ef1674f85bf480 [term 0 FOLLOWER]: Advancing to term 1
I20260924 21:08:33.432734 8589 raft_consensus.cc:504] T 00000000000000000000000000000000 P e592d546539745c395ef1674f85bf480 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e592d546539745c395ef1674f85bf480" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34181 } }
I20260924 21:08:33.432834 8589 leader_election.cc:302] T 00000000000000000000000000000000 P e592d546539745c395ef1674f85bf480 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e592d546539745c395ef1674f85bf480; no voters:
I20260924 21:08:33.432976 8589 leader_election.cc:288] T 00000000000000000000000000000000 P e592d546539745c395ef1674f85bf480 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 21:08:33.433008 8592 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e592d546539745c395ef1674f85bf480 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 21:08:33.433139 8592 raft_consensus.cc:686] T 00000000000000000000000000000000 P e592d546539745c395ef1674f85bf480 [term 1 LEADER]: Becoming Leader. State: Replica: e592d546539745c395ef1674f85bf480, State: Running, Role: LEADER
I20260924 21:08:33.433193 8589 sys_catalog.cc:564] T 00000000000000000000000000000000 P e592d546539745c395ef1674f85bf480 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 21:08:33.433252 8592 consensus_queue.cc:237] T 00000000000000000000000000000000 P e592d546539745c395ef1674f85bf480 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e592d546539745c395ef1674f85bf480" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34181 } }
I20260924 21:08:33.433593 8593 sys_catalog.cc:455] T 00000000000000000000000000000000 P e592d546539745c395ef1674f85bf480 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e592d546539745c395ef1674f85bf480" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e592d546539745c395ef1674f85bf480" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34181 } } }
I20260924 21:08:33.433595 8594 sys_catalog.cc:455] T 00000000000000000000000000000000 P e592d546539745c395ef1674f85bf480 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e592d546539745c395ef1674f85bf480. Latest consensus state: current_term: 1 leader_uuid: "e592d546539745c395ef1674f85bf480" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e592d546539745c395ef1674f85bf480" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34181 } } }
I20260924 21:08:33.433707 8593 sys_catalog.cc:458] T 00000000000000000000000000000000 P e592d546539745c395ef1674f85bf480 [sys.catalog]: This master's current role is: LEADER
I20260924 21:08:33.433715 8594 sys_catalog.cc:458] T 00000000000000000000000000000000 P e592d546539745c395ef1674f85bf480 [sys.catalog]: This master's current role is: LEADER
I20260924 21:08:33.433919 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAzNNjS/build/release/bin/kudu as pid 8521
I20260924 21:08:33.434002 8599 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 21:08:33.434036 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790284076800039-392-0/minicluster-data/master-0/wal/instance
I20260924 21:08:33.434262 392 external_mini_cluster.cc:1428] Setting key 911dcbdaee687a2d4996794d68cb1089
I20260924 21:08:33.434437 8599 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 21:08:33.435806 8599 catalog_manager.cc:1347] Generated new cluster ID: 6f517a938396480a808bd65c401806d3
I20260924 21:08:33.435866 8599 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 24 21:08:33 dist-test-slave-c7tk krb5kdc[8508](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284113, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 21:08:33.444581 8605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:33.434672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37218 (local address 127.0.98.62:34181)
0924 21:08:33.435206 (+ 534us) server_negotiation.cc:206] Beginning negotiation
0924 21:08:33.435212 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 21:08:33.435305 (+ 93us) server_negotiation.cc:407] Connection header received
0924 21:08:33.435420 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:08:33.435426 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:08:33.435491 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:08:33.435593 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:08:33.436265 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:33.437462 (+ 1197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:33.438382 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:33.438946 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:33.439049 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:33.441604 (+ 2555us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 21:08:33.441607 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 21:08:33.441607 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 21:08:33.441616 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 21:08:33.443177 (+ 1561us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:08:33.443528 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:08:33.443530 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:08:33.443530 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:08:33.443609 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:08:33.443788 (+ 179us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:08:33.443788 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:08:33.443789 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:08:33.444011 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 21:08:33.444127 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0924 21:08:33.444268 (+ 141us) server_negotiation.cc:299] Negotiation successful
0924 21:08:33.444380 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":498,"thread_start_us":64,"threads_started":1}
I20260924 21:08:33.446305 8599 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 21:08:33.446756 8599 catalog_manager.cc:1504] Loading token signing keys...
I20260924 21:08:33.450382 8599 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e592d546539745c395ef1674f85bf480: Generated new TSK 0
I20260924 21:08:33.450538 8599 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790284076800039-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790284076800039-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260924 21:08:33.474808 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAzNNjS/build/release/bin/kudu
/tmp/dist-test-taskAzNNjS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790284076800039-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790284076800039-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790284076800039-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790284076800039-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34181
--keytab_file=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790284076800039-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41267
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790284076800039-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790284076800039-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790284076800039-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 21:08:33.545433 8615 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 21:08:33.545615 8615 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 21:08:33.545636 8615 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 21:08:33.546860 8615 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 21:08:33.546902 8615 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 21:08:33.546952 8615 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 21:08:33.548306 8615 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41267
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790284076800039-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790284076800039-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790284076800039-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790284076800039-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34181
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8615
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790284076800039-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 19:49:06 UTC on ccycloud.quasar-vhqjoc.root.comops.site
build id 4670864
I20260924 21:08:33.548514 8615 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 21:08:33.548718 8615 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 21:08:33.553054 8621 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 21:08:33.553068 8620 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 21:08:33.553215 8615 server_base.cc:1034] running on GCE node
W20260924 21:08:33.553100 8623 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 21:08:33.553450 8615 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 21:08:33.553661 8615 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 21:08:33.554824 8615 hybrid_clock.cc:648] HybridClock initialized: now 1790284113554822 us; error 32 us; skew 500 ppm
Sep 24 21:08:33 dist-test-slave-c7tk krb5kdc[8508](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284113, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 21:08:33.556955 8615 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 21:08:33.557327 8615 webserver.cc:460] Webserver started at http://127.0.98.1:40763/ using document root <none> and password file <none>
I20260924 21:08:33.557542 8615 fs_manager.cc:359] Metadata directory not provided
I20260924 21:08:33.557602 8615 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 21:08:33.557716 8615 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 21:08:33.558646 8615 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790284076800039-392-0/minicluster-data/ts-0/data/instance:
uuid: "34cb1009bc3b458eaa58fdf27b5ee594"
format_stamp: "Formatted at 2026-09-24 21:08:33 on dist-test-slave-c7tk"
server_key: "3d35e4d4c9e9155917928373d622e9ea"
server_key_iv: "ff2e7484529d4734a18cd4ce7f4e0c31"
server_key_version: "encryptionkey@0"
I20260924 21:08:33.558964 8615 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790284076800039-392-0/minicluster-data/ts-0/wal/instance:
uuid: "34cb1009bc3b458eaa58fdf27b5ee594"
format_stamp: "Formatted at 2026-09-24 21:08:33 on dist-test-slave-c7tk"
server_key: "3d35e4d4c9e9155917928373d622e9ea"
server_key_iv: "ff2e7484529d4734a18cd4ce7f4e0c31"
server_key_version: "encryptionkey@0"
I20260924 21:08:33.560171 8615 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260924 21:08:33.560837 8630 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 21:08:33.560972 8615 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 21:08:33.561053 8615 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790284076800039-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790284076800039-392-0/minicluster-data/ts-0/wal
uuid: "34cb1009bc3b458eaa58fdf27b5ee594"
format_stamp: "Formatted at 2026-09-24 21:08:33 on dist-test-slave-c7tk"
server_key: "3d35e4d4c9e9155917928373d622e9ea"
server_key_iv: "ff2e7484529d4734a18cd4ce7f4e0c31"
server_key_version: "encryptionkey@0"
I20260924 21:08:33.561123 8615 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790284076800039-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790284076800039-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790284076800039-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 21:08:33.570827 8615 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 21:08:33.572415 8615 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 21:08:33.572563 8615 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 21:08:33.572762 8615 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 21:08:33.573041 8615 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 21:08:33.573091 8615 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 21:08:33.573130 8615 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 21:08:33.573153 8615 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 21:08:33.578545 8615 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:44135
I20260924 21:08:33.578608 8743 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:44135 every 8 connection(s)
I20260924 21:08:33.578887 8615 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790284076800039-392-0/minicluster-data/ts-0/data/info.pb
Sep 24 21:08:33 dist-test-slave-c7tk krb5kdc[8508](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284113, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 21:08:33.587110 8605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:33.579696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:60325 (local address 127.0.98.62:34181)
0924 21:08:33.579737 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0924 21:08:33.579739 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 21:08:33.579930 (+ 191us) server_negotiation.cc:407] Connection header received
0924 21:08:33.580814 (+ 884us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:08:33.580816 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:08:33.580827 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:08:33.580872 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:08:33.581404 (+ 532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:33.581923 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:33.582881 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:33.583087 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:33.583157 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:33.584881 (+ 1724us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 21:08:33.584882 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 21:08:33.584883 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 21:08:33.584890 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 21:08:33.585933 (+ 1043us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:08:33.586269 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:08:33.586270 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:08:33.586271 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:08:33.586326 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:08:33.586540 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:08:33.586541 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:08:33.586541 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:08:33.586645 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 21:08:33.586710 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0924 21:08:33.586956 (+ 246us) server_negotiation.cc:299] Negotiation successful
0924 21:08:33.587029 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 21:08:33.587261 8746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:33.579665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34181 (local address 127.0.98.1:60325)
0924 21:08:33.579826 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0924 21:08:33.579856 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0924 21:08:33.580713 (+ 857us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 21:08:33.580927 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 21:08:33.580931 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 21:08:33.581165 (+ 234us) client_negotiation.cc:189] Negotiated authn=SASL
0924 21:08:33.581317 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:33.581322 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:33.582023 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:33.582024 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:33.582772 (+ 748us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:33.582774 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:33.583160 (+ 386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:33.583161 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:33.583278 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:33.583691 (+ 413us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 21:08:33.583698 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0924 21:08:33.584780 (+ 1082us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 21:08:33.586008 (+ 1228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 21:08:33.586014 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 21:08:33.586015 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 21:08:33.586193 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 21:08:33.586384 (+ 191us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 21:08:33.586385 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 21:08:33.586385 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 21:08:33.586477 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 21:08:33.586702 (+ 225us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 21:08:33.586707 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 21:08:33.586842 (+ 135us) client_negotiation.cc:769] Sending connection context
0924 21:08:33.586937 (+ 95us) client_negotiation.cc:240] Negotiation successful
0924 21:08:33.587067 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":74,"threads_started":1}
I20260924 21:08:33.587739 8744 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34181
I20260924 21:08:33.587805 8744 heartbeater.cc:464] Registering TS with master...
I20260924 21:08:33.587961 8744 heartbeater.cc:511] Master 127.0.98.62:34181 requested a full tablet report, sending...
I20260924 21:08:33.588384 8553 ts_manager.cc:194] Registered new tserver with Master: 34cb1009bc3b458eaa58fdf27b5ee594 (127.0.98.1:44135)
I20260924 21:08:33.588601 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAzNNjS/build/release/bin/kudu as pid 8615
I20260924 21:08:33.588691 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790284076800039-392-0/minicluster-data/ts-0/wal/instance
I20260924 21:08:33.588876 392 external_mini_cluster.cc:1428] Setting key 171fcefee3c33f733db8a959fc08c3c0
I20260924 21:08:33.589073 8553 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:60325
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790284076800039-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790284076800039-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260924 21:08:33.610126 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAzNNjS/build/release/bin/kudu
/tmp/dist-test-taskAzNNjS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790284076800039-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790284076800039-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790284076800039-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790284076800039-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34181
--keytab_file=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790284076800039-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41267
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790284076800039-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790284076800039-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790284076800039-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 21:08:33.682126 8749 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 21:08:33.682335 8749 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 21:08:33.682372 8749 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 21:08:33.683449 8749 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 21:08:33.683509 8749 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 21:08:33.683547 8749 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 21:08:33.684821 8749 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41267
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790284076800039-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790284076800039-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790284076800039-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790284076800039-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34181
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8749
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790284076800039-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 19:49:06 UTC on ccycloud.quasar-vhqjoc.root.comops.site
build id 4670864
I20260924 21:08:33.685036 8749 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 21:08:33.685221 8749 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 21:08:33.689330 8749 server_base.cc:1034] running on GCE node
W20260924 21:08:33.689383 8754 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 21:08:33.689414 8757 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 21:08:33.689642 8755 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 21:08:33.689900 8749 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 21:08:33.690163 8749 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 21:08:33.691311 8749 hybrid_clock.cc:648] HybridClock initialized: now 1790284113691302 us; error 43 us; skew 500 ppm
Sep 24 21:08:33 dist-test-slave-c7tk krb5kdc[8508](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284113, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 21:08:33.702615 8749 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 21:08:33.702977 8749 webserver.cc:460] Webserver started at http://127.0.98.2:43493/ using document root <none> and password file <none>
I20260924 21:08:33.703159 8749 fs_manager.cc:359] Metadata directory not provided
I20260924 21:08:33.703218 8749 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 21:08:33.703349 8749 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 21:08:33.704267 8749 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790284076800039-392-0/minicluster-data/ts-1/data/instance:
uuid: "c4ad82d1f8b84c1fa92afa6a1c44839f"
format_stamp: "Formatted at 2026-09-24 21:08:33 on dist-test-slave-c7tk"
server_key: "d258f626c31ac7bec214578f350dd607"
server_key_iv: "e7ec44292d0b5d6bf6755aa5401ee03a"
server_key_version: "encryptionkey@0"
I20260924 21:08:33.704571 8749 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790284076800039-392-0/minicluster-data/ts-1/wal/instance:
uuid: "c4ad82d1f8b84c1fa92afa6a1c44839f"
format_stamp: "Formatted at 2026-09-24 21:08:33 on dist-test-slave-c7tk"
server_key: "d258f626c31ac7bec214578f350dd607"
server_key_iv: "e7ec44292d0b5d6bf6755aa5401ee03a"
server_key_version: "encryptionkey@0"
I20260924 21:08:33.705636 8749 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 21:08:33.706391 8764 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 21:08:33.706547 8749 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 21:08:33.706650 8749 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790284076800039-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790284076800039-392-0/minicluster-data/ts-1/wal
uuid: "c4ad82d1f8b84c1fa92afa6a1c44839f"
format_stamp: "Formatted at 2026-09-24 21:08:33 on dist-test-slave-c7tk"
server_key: "d258f626c31ac7bec214578f350dd607"
server_key_iv: "e7ec44292d0b5d6bf6755aa5401ee03a"
server_key_version: "encryptionkey@0"
I20260924 21:08:33.706732 8749 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790284076800039-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790284076800039-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790284076800039-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 21:08:33.712260 8749 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 21:08:33.713783 8749 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 21:08:33.713961 8749 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 21:08:33.714157 8749 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 21:08:33.714419 8749 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 21:08:33.714470 8749 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 21:08:33.714506 8749 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 21:08:33.714532 8749 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 21:08:33.719712 8749 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:40221
I20260924 21:08:33.719743 8877 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:40221 every 8 connection(s)
I20260924 21:08:33.720026 8749 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790284076800039-392-0/minicluster-data/ts-1/data/info.pb
I20260924 21:08:33.723879 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAzNNjS/build/release/bin/kudu as pid 8749
I20260924 21:08:33.723979 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790284076800039-392-0/minicluster-data/ts-1/wal/instance
I20260924 21:08:33.724123 392 external_mini_cluster.cc:1428] Setting key f872dc0ce930ed94e83e7da51f27fc2d
Sep 24 21:08:33 dist-test-slave-c7tk krb5kdc[8508](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284113, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 21:08:33.727995 8605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:33.720686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:59647 (local address 127.0.98.62:34181)
0924 21:08:33.720737 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0924 21:08:33.720738 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 21:08:33.720909 (+ 171us) server_negotiation.cc:407] Connection header received
0924 21:08:33.721666 (+ 757us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:08:33.721667 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:08:33.721678 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:08:33.721727 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:08:33.722255 (+ 528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:33.722675 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:33.723505 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:33.723671 (+ 166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:33.723734 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:33.725457 (+ 1723us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 21:08:33.725458 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 21:08:33.725459 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 21:08:33.725465 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 21:08:33.726803 (+ 1338us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:08:33.727206 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:08:33.727207 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:08:33.727207 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:08:33.727254 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:08:33.727492 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:08:33.727493 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:08:33.727493 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:08:33.727576 (+ 83us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 21:08:33.727619 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0924 21:08:33.727886 (+ 267us) server_negotiation.cc:299] Negotiation successful
0924 21:08:33.727942 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 21:08:33.728215 8880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:33.720635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34181 (local address 127.0.98.2:59647)
0924 21:08:33.720815 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0924 21:08:33.720843 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0924 21:08:33.721573 (+ 730us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 21:08:33.721776 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 21:08:33.721780 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 21:08:33.722023 (+ 243us) client_negotiation.cc:189] Negotiated authn=SASL
0924 21:08:33.722166 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:33.722169 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:33.722753 (+ 584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:33.722754 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:33.723426 (+ 672us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:33.723427 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:33.723767 (+ 340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:33.723768 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:33.723865 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:33.724282 (+ 417us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 21:08:33.724289 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0924 21:08:33.725375 (+ 1086us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 21:08:33.726881 (+ 1506us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 21:08:33.726887 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 21:08:33.726888 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 21:08:33.727137 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 21:08:33.727320 (+ 183us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 21:08:33.727321 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 21:08:33.727322 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 21:08:33.727438 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 21:08:33.727630 (+ 192us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 21:08:33.727634 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 21:08:33.727760 (+ 126us) client_negotiation.cc:769] Sending connection context
0924 21:08:33.727863 (+ 103us) client_negotiation.cc:240] Negotiation successful
0924 21:08:33.728007 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":102,"threads_started":1}
I20260924 21:08:33.728667 8878 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34181
I20260924 21:08:33.728749 8878 heartbeater.cc:464] Registering TS with master...
I20260924 21:08:33.728901 8878 heartbeater.cc:511] Master 127.0.98.62:34181 requested a full tablet report, sending...
I20260924 21:08:33.729208 8552 ts_manager.cc:194] Registered new tserver with Master: c4ad82d1f8b84c1fa92afa6a1c44839f (127.0.98.2:40221)
I20260924 21:08:33.729568 8552 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:59647
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790284076800039-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790284076800039-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260924 21:08:33.745988 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAzNNjS/build/release/bin/kudu
/tmp/dist-test-taskAzNNjS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790284076800039-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790284076800039-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790284076800039-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790284076800039-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34181
--keytab_file=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790284076800039-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41267
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790284076800039-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790284076800039-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790284076800039-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 21:08:33.815366 8883 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 21:08:33.815634 8883 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 21:08:33.815682 8883 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 21:08:33.816653 8883 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 21:08:33.816687 8883 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 21:08:33.816708 8883 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 21:08:33.817977 8883 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41267
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790284076800039-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790284076800039-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790284076800039-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790284076800039-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34181
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8883
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790284076800039-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 19:49:06 UTC on ccycloud.quasar-vhqjoc.root.comops.site
build id 4670864
I20260924 21:08:33.818125 8883 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 21:08:33.818308 8883 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 21:08:33.822654 8891 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 21:08:33.822667 8889 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 21:08:33.822752 8888 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 21:08:33.822944 8883 server_base.cc:1034] running on GCE node
I20260924 21:08:33.823128 8883 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 21:08:33.823347 8883 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 21:08:33.824482 8883 hybrid_clock.cc:648] HybridClock initialized: now 1790284113824467 us; error 25 us; skew 500 ppm
Sep 24 21:08:33 dist-test-slave-c7tk krb5kdc[8508](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284113, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 21:08:33.826603 8883 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 21:08:33.826951 8883 webserver.cc:460] Webserver started at http://127.0.98.3:43653/ using document root <none> and password file <none>
I20260924 21:08:33.827129 8883 fs_manager.cc:359] Metadata directory not provided
I20260924 21:08:33.827184 8883 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 21:08:33.827308 8883 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 21:08:33.828142 8883 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790284076800039-392-0/minicluster-data/ts-2/data/instance:
uuid: "85b113ef14db4e9aad1e42eadced86a8"
format_stamp: "Formatted at 2026-09-24 21:08:33 on dist-test-slave-c7tk"
server_key: "6f1565d771fc777dc6917a75def85c34"
server_key_iv: "f20fc01f0b8b72f9acb12bb9888ca3b9"
server_key_version: "encryptionkey@0"
I20260924 21:08:33.828457 8883 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790284076800039-392-0/minicluster-data/ts-2/wal/instance:
uuid: "85b113ef14db4e9aad1e42eadced86a8"
format_stamp: "Formatted at 2026-09-24 21:08:33 on dist-test-slave-c7tk"
server_key: "6f1565d771fc777dc6917a75def85c34"
server_key_iv: "f20fc01f0b8b72f9acb12bb9888ca3b9"
server_key_version: "encryptionkey@0"
I20260924 21:08:33.829577 8883 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 21:08:33.830323 8898 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 21:08:33.830492 8883 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 21:08:33.830585 8883 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790284076800039-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790284076800039-392-0/minicluster-data/ts-2/wal
uuid: "85b113ef14db4e9aad1e42eadced86a8"
format_stamp: "Formatted at 2026-09-24 21:08:33 on dist-test-slave-c7tk"
server_key: "6f1565d771fc777dc6917a75def85c34"
server_key_iv: "f20fc01f0b8b72f9acb12bb9888ca3b9"
server_key_version: "encryptionkey@0"
I20260924 21:08:33.830670 8883 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790284076800039-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790284076800039-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790284076800039-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 21:08:33.846614 8883 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 21:08:33.848429 8883 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 21:08:33.848567 8883 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 21:08:33.848747 8883 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 21:08:33.848995 8883 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 21:08:33.849040 8883 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 21:08:33.849067 8883 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 21:08:33.849090 8883 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 21:08:33.854271 8883 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:44627
I20260924 21:08:33.854329 9011 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:44627 every 8 connection(s)
I20260924 21:08:33.854594 8883 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790284076800039-392-0/minicluster-data/ts-2/data/info.pb
I20260924 21:08:33.859452 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAzNNjS/build/release/bin/kudu as pid 8883
I20260924 21:08:33.859570 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790284076800039-392-0/minicluster-data/ts-2/wal/instance
Sep 24 21:08:33 dist-test-slave-c7tk krb5kdc[8508](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284113, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 21:08:33.859760 392 external_mini_cluster.cc:1428] Setting key 453f4ffd5bd65d57ecbb505ff4d2761e
I20260924 21:08:33.862569 8605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:33.855234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:42719 (local address 127.0.98.62:34181)
0924 21:08:33.855272 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0924 21:08:33.855274 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 21:08:33.855433 (+ 159us) server_negotiation.cc:407] Connection header received
0924 21:08:33.856190 (+ 757us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:08:33.856191 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:08:33.856203 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:08:33.856247 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:08:33.856740 (+ 493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:33.857180 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:33.858096 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:33.858266 (+ 170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:33.858339 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:33.860161 (+ 1822us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 21:08:33.860162 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 21:08:33.860162 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 21:08:33.860170 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 21:08:33.861411 (+ 1241us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:08:33.861741 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:08:33.861742 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:08:33.861743 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:08:33.861789 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:08:33.862027 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:08:33.862027 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:08:33.862028 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:08:33.862130 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 21:08:33.862195 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0924 21:08:33.862428 (+ 233us) server_negotiation.cc:299] Negotiation successful
0924 21:08:33.862495 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 21:08:33.862699 9014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:33.855185 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34181 (local address 127.0.98.3:42719)
0924 21:08:33.855337 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0924 21:08:33.855364 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0924 21:08:33.856093 (+ 729us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 21:08:33.856302 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 21:08:33.856305 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 21:08:33.856513 (+ 208us) client_negotiation.cc:189] Negotiated authn=SASL
0924 21:08:33.856654 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:33.856657 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:33.857261 (+ 604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:33.857262 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:33.858020 (+ 758us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:33.858022 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:33.858352 (+ 330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:33.858352 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:33.858461 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:33.858781 (+ 320us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 21:08:33.858787 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0924 21:08:33.860081 (+ 1294us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 21:08:33.861484 (+ 1403us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 21:08:33.861489 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 21:08:33.861492 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 21:08:33.861674 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 21:08:33.861867 (+ 193us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 21:08:33.861868 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 21:08:33.861868 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 21:08:33.861962 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 21:08:33.862186 (+ 224us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 21:08:33.862192 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 21:08:33.862307 (+ 115us) client_negotiation.cc:769] Sending connection context
0924 21:08:33.862399 (+ 92us) client_negotiation.cc:240] Negotiation successful
0924 21:08:33.862520 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":56,"threads_started":1}
I20260924 21:08:33.863108 9012 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34181
I20260924 21:08:33.863199 9012 heartbeater.cc:464] Registering TS with master...
I20260924 21:08:33.863343 9012 heartbeater.cc:511] Master 127.0.98.62:34181 requested a full tablet report, sending...
I20260924 21:08:33.863652 8552 ts_manager.cc:194] Registered new tserver with Master: 85b113ef14db4e9aad1e42eadced86a8 (127.0.98.3:44627)
I20260924 21:08:33.863970 8552 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:42719
I20260924 21:08:33.871753 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790284076800039-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790284076800039-392-0/krb5kdc/oryx_127.0.98.1.keytab.
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790284076800039-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790284076800039-392-0/krb5kdc/oryx_127.0.98.2.keytab.
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790284076800039-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790284076800039-392-0/krb5kdc/oryx_127.0.98.3.keytab.
I20260924 21:08:33.930315 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAzNNjS/build/release/bin/kudu with pid 8615
I20260924 21:08:33.934743 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAzNNjS/build/release/bin/kudu with pid 8749
I20260924 21:08:33.938830 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAzNNjS/build/release/bin/kudu with pid 8883
I20260924 21:08:33.942590 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAzNNjS/build/release/bin/kudu with pid 8521
I20260924 21:08:33.946779 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAzNNjS/build/release/bin/kudu
/tmp/dist-test-taskAzNNjS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790284076800039-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790284076800039-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790284076800039-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790284076800039-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34181
--webserver_interface=127.0.98.62
--webserver_port=41211
--keytab_file=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790284076800039-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41267
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34181
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790284076800039-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790284076800039-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790284076800039-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 21:08:34.016173 9021 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 21:08:34.016355 9021 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 21:08:34.016387 9021 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 21:08:34.017364 9021 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 21:08:34.017422 9021 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 21:08:34.017443 9021 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 21:08:34.017467 9021 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 21:08:34.017486 9021 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 21:08:34.018764 9021 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41267
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790284076800039-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790284076800039-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34181
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790284076800039-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34181
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790284076800039-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=41211
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9021
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790284076800039-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 19:49:06 UTC on ccycloud.quasar-vhqjoc.root.comops.site
build id 4670864
I20260924 21:08:34.018992 9021 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 21:08:34.019204 9021 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 21:08:34.023540 9029 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 21:08:34.023576 9021 server_base.cc:1034] running on GCE node
W20260924 21:08:34.023523 9027 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 21:08:34.023510 9026 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 21:08:34.023944 9021 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 21:08:34.024147 9021 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 21:08:34.025307 9021 hybrid_clock.cc:648] HybridClock initialized: now 1790284114025297 us; error 25 us; skew 500 ppm
Sep 24 21:08:34 dist-test-slave-c7tk krb5kdc[8508](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284114, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 21:08:34.027415 9021 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 21:08:34.027814 9021 webserver.cc:460] Webserver started at http://127.0.98.62:41211/ using document root <none> and password file <none>
I20260924 21:08:34.028028 9021 fs_manager.cc:359] Metadata directory not provided
I20260924 21:08:34.028108 9021 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 21:08:34.029263 9021 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 21:08:34.029896 9036 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 21:08:34.030054 9021 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 21:08:34.030133 9021 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790284076800039-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790284076800039-392-0/minicluster-data/master-0/wal
uuid: "e592d546539745c395ef1674f85bf480"
format_stamp: "Formatted at 2026-09-24 21:08:33 on dist-test-slave-c7tk"
server_key: "bb37e1f0c442500763bc536742e13aa3"
server_key_iv: "5e21e257844f93a6e2c18893f16e04af"
server_key_version: "encryptionkey@0"
I20260924 21:08:34.030387 9021 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790284076800039-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790284076800039-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790284076800039-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 21:08:34.040743 9021 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 21:08:34.042709 9021 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 21:08:34.042843 9021 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 21:08:34.047358 9021 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34181
I20260924 21:08:34.047384 9088 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34181 every 8 connection(s)
I20260924 21:08:34.047881 9021 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790284076800039-392-0/minicluster-data/master-0/data/info.pb
I20260924 21:08:34.048869 9089 sys_catalog.cc:263] Verifying existing consensus state
I20260924 21:08:34.049396 9089 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e592d546539745c395ef1674f85bf480: Bootstrap starting.
I20260924 21:08:34.050257 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAzNNjS/build/release/bin/kudu as pid 9021
I20260924 21:08:34.050552 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAzNNjS/build/release/bin/kudu
/tmp/dist-test-taskAzNNjS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790284076800039-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790284076800039-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790284076800039-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790284076800039-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:44135
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:34181
--webserver_port=40763
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790284076800039-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41267
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.1
--keytab_file=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790284076800039-392-0/krb5kdc/oryx_127.0.98.1.keytab with env {KRB5CCNAME=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790284076800039-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790284076800039-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790284076800039-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 21:08:34.057081 9089 log.cc:824] T 00000000000000000000000000000000 P e592d546539745c395ef1674f85bf480: Log is configured to *not* fsync() on all Append() calls
I20260924 21:08:34.058475 9089 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e592d546539745c395ef1674f85bf480: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 21:08:34.058713 9089 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e592d546539745c395ef1674f85bf480: Bootstrap complete.
I20260924 21:08:34.060846 9089 raft_consensus.cc:348] T 00000000000000000000000000000000 P e592d546539745c395ef1674f85bf480 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e592d546539745c395ef1674f85bf480" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34181 } }
I20260924 21:08:34.061074 9089 raft_consensus.cc:729] T 00000000000000000000000000000000 P e592d546539745c395ef1674f85bf480 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e592d546539745c395ef1674f85bf480, State: Initialized, Role: FOLLOWER
I20260924 21:08:34.061257 9089 consensus_queue.cc:260] T 00000000000000000000000000000000 P e592d546539745c395ef1674f85bf480 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e592d546539745c395ef1674f85bf480" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34181 } }
I20260924 21:08:34.061349 9089 raft_consensus.cc:388] T 00000000000000000000000000000000 P e592d546539745c395ef1674f85bf480 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 21:08:34.061398 9089 raft_consensus.cc:482] T 00000000000000000000000000000000 P e592d546539745c395ef1674f85bf480 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 21:08:34.061439 9089 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e592d546539745c395ef1674f85bf480 [term 1 FOLLOWER]: Advancing to term 2
I20260924 21:08:34.062356 9089 raft_consensus.cc:504] T 00000000000000000000000000000000 P e592d546539745c395ef1674f85bf480 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e592d546539745c395ef1674f85bf480" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34181 } }
I20260924 21:08:34.062461 9089 leader_election.cc:302] T 00000000000000000000000000000000 P e592d546539745c395ef1674f85bf480 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e592d546539745c395ef1674f85bf480; no voters:
I20260924 21:08:34.062629 9089 leader_election.cc:288] T 00000000000000000000000000000000 P e592d546539745c395ef1674f85bf480 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260924 21:08:34.062656 9093 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e592d546539745c395ef1674f85bf480 [term 2 FOLLOWER]: Leader election won for term 2
I20260924 21:08:34.062757 9093 raft_consensus.cc:686] T 00000000000000000000000000000000 P e592d546539745c395ef1674f85bf480 [term 2 LEADER]: Becoming Leader. State: Replica: e592d546539745c395ef1674f85bf480, State: Running, Role: LEADER
I20260924 21:08:34.062831 9093 consensus_queue.cc:237] T 00000000000000000000000000000000 P e592d546539745c395ef1674f85bf480 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e592d546539745c395ef1674f85bf480" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34181 } }
I20260924 21:08:34.062863 9089 sys_catalog.cc:564] T 00000000000000000000000000000000 P e592d546539745c395ef1674f85bf480 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 21:08:34.063145 9095 sys_catalog.cc:455] T 00000000000000000000000000000000 P e592d546539745c395ef1674f85bf480 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e592d546539745c395ef1674f85bf480. Latest consensus state: current_term: 2 leader_uuid: "e592d546539745c395ef1674f85bf480" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e592d546539745c395ef1674f85bf480" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34181 } } }
I20260924 21:08:34.063182 9094 sys_catalog.cc:455] T 00000000000000000000000000000000 P e592d546539745c395ef1674f85bf480 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "e592d546539745c395ef1674f85bf480" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e592d546539745c395ef1674f85bf480" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34181 } } }
I20260924 21:08:34.063323 9095 sys_catalog.cc:458] T 00000000000000000000000000000000 P e592d546539745c395ef1674f85bf480 [sys.catalog]: This master's current role is: LEADER
I20260924 21:08:34.063338 9094 sys_catalog.cc:458] T 00000000000000000000000000000000 P e592d546539745c395ef1674f85bf480 [sys.catalog]: This master's current role is: LEADER
I20260924 21:08:34.063637 9102 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 21:08:34.064066 9102 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 21:08:34.064193 9102 catalog_manager.cc:1259] Loaded cluster ID: 6f517a938396480a808bd65c401806d3
I20260924 21:08:34.064388 9102 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 21:08:34.065073 9102 catalog_manager.cc:1504] Loading token signing keys...
I20260924 21:08:34.065258 9102 catalog_manager.cc:5879] T 00000000000000000000000000000000 P e592d546539745c395ef1674f85bf480: Loaded TSK: 0
I20260924 21:08:34.065443 9102 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260924 21:08:34.135934 9091 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 21:08:34.136089 9091 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 21:08:34.136107 9091 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 21:08:34.137072 9091 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 21:08:34.137108 9091 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 21:08:34.137130 9091 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 21:08:34.138345 9091 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41267
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790284076800039-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790284076800039-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790284076800039-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:44135
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790284076800039-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=40763
--tserver_master_addrs=127.0.98.62:34181
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9091
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790284076800039-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 19:49:06 UTC on ccycloud.quasar-vhqjoc.root.comops.site
build id 4670864
I20260924 21:08:34.138531 9091 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 21:08:34.138731 9091 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 21:08:34.143777 9118 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 21:08:34.143802 9115 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 21:08:34.143774 9116 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 21:08:34.144167 9091 server_base.cc:1034] running on GCE node
I20260924 21:08:34.144347 9091 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 21:08:34.144603 9091 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 21:08:34.145769 9091 hybrid_clock.cc:648] HybridClock initialized: now 1790284114145751 us; error 29 us; skew 500 ppm
Sep 24 21:08:34 dist-test-slave-c7tk krb5kdc[8508](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284114, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 21:08:34.148188 9091 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260924 21:08:34.148590 9091 webserver.cc:460] Webserver started at http://127.0.98.1:40763/ using document root <none> and password file <none>
I20260924 21:08:34.148790 9091 fs_manager.cc:359] Metadata directory not provided
I20260924 21:08:34.148838 9091 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 21:08:34.150111 9091 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 21:08:34.150756 9125 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 21:08:34.150931 9091 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 21:08:34.151010 9091 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790284076800039-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790284076800039-392-0/minicluster-data/ts-0/wal
uuid: "34cb1009bc3b458eaa58fdf27b5ee594"
format_stamp: "Formatted at 2026-09-24 21:08:33 on dist-test-slave-c7tk"
server_key: "3d35e4d4c9e9155917928373d622e9ea"
server_key_iv: "ff2e7484529d4734a18cd4ce7f4e0c31"
server_key_version: "encryptionkey@0"
I20260924 21:08:34.151268 9091 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790284076800039-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790284076800039-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790284076800039-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 21:08:34.170089 9091 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 21:08:34.172119 9091 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 21:08:34.172260 9091 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 21:08:34.172484 9091 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 21:08:34.172742 9091 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 21:08:34.172787 9091 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 21:08:34.172819 9091 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 21:08:34.172837 9091 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 21:08:34.179296 9091 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:44135
I20260924 21:08:34.179328 9238 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:44135 every 8 connection(s)
I20260924 21:08:34.179680 9091 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790284076800039-392-0/minicluster-data/ts-0/data/info.pb
I20260924 21:08:34.184974 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAzNNjS/build/release/bin/kudu as pid 9091
I20260924 21:08:34.185271 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAzNNjS/build/release/bin/kudu
/tmp/dist-test-taskAzNNjS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790284076800039-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790284076800039-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790284076800039-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790284076800039-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:40221
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:34181
--webserver_port=43493
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790284076800039-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41267
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.2
--keytab_file=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790284076800039-392-0/krb5kdc/oryx_127.0.98.2.keytab with env {KRB5CCNAME=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790284076800039-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790284076800039-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790284076800039-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 21:08:34 dist-test-slave-c7tk krb5kdc[8508](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 21:08:34 dist-test-slave-c7tk krb5kdc[8508](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 21:08:34.187543 9241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:34.180486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34181 (local address 127.0.98.1:48431)
0924 21:08:34.180653 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0924 21:08:34.180709 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0924 21:08:34.181151 (+ 442us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 21:08:34.181544 (+ 393us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 21:08:34.181547 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 21:08:34.182098 (+ 551us) client_negotiation.cc:189] Negotiated authn=SASL
0924 21:08:34.182250 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:34.182254 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:34.183345 (+ 1091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:34.183346 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:34.184081 (+ 735us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:34.184083 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:34.184642 (+ 559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:34.184643 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:34.184750 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:34.185219 (+ 469us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 21:08:34.185228 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 21:08:34.187375 (+ 2147us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34181: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":77,"threads_started":1}
W20260924 21:08:34.187589 9242 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 21:08:34.180546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:48431 (local address 127.0.98.62:34181)
0924 21:08:34.180744 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0924 21:08:34.180752 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 21:08:34.180793 (+ 41us) server_negotiation.cc:407] Connection header received
0924 21:08:34.181331 (+ 538us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:08:34.181336 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:08:34.181381 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:08:34.181497 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:08:34.182355 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:34.183225 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:34.184188 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:34.184551 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:34.184661 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:34.187466 (+ 2805us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:48431: BlockingRecv error: recv got EOF from 127.0.98.1:48431 (error 108)
Metrics: {"server-negotiator.queue_time_us":158,"thread_start_us":91,"threads_started":1}
W20260924 21:08:34.187965 9239 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:34181 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:34181: Client connection negotiation failed: client connection to 127.0.98.62:34181: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Sep 24 21:08:34 dist-test-slave-c7tk krb5kdc[8508](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 21:08:34 dist-test-slave-c7tk krb5kdc[8508](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 21:08:34.191917 9241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:34.188195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34181 (local address 127.0.98.1:48125)
0924 21:08:34.188266 (+ 71us) negotiation.cc:107] Waiting for socket to connect
0924 21:08:34.188275 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 21:08:34.188313 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 21:08:34.188414 (+ 101us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 21:08:34.188414 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 21:08:34.188526 (+ 112us) client_negotiation.cc:189] Negotiated authn=SASL
0924 21:08:34.188559 (+ 33us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:34.188560 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:34.189077 (+ 517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:34.189078 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:34.189500 (+ 422us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:34.189501 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:34.189907 (+ 406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:34.189908 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:34.189972 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:34.190345 (+ 373us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 21:08:34.190348 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0924 21:08:34.191854 (+ 1506us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34181: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":61}
W20260924 21:08:34.191962 9242 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 21:08:34.188237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:48125 (local address 127.0.98.62:34181)
0924 21:08:34.188268 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0924 21:08:34.188270 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 21:08:34.188302 (+ 32us) server_negotiation.cc:407] Connection header received
0924 21:08:34.188354 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:08:34.188355 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:08:34.188364 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:08:34.188411 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:08:34.188623 (+ 212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:34.189011 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:34.189602 (+ 591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:34.189842 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:34.189907 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:34.191909 (+ 2002us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:48125: BlockingRecv error: recv got EOF from 127.0.98.1:48125 (error 108)
Metrics: {"server-negotiator.queue_time_us":13}
Sep 24 21:08:34 dist-test-slave-c7tk krb5kdc[8508](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 21:08:34 dist-test-slave-c7tk krb5kdc[8508](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 21:08:34.195719 9241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:34.192356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34181 (local address 127.0.98.1:38367)
0924 21:08:34.192388 (+ 32us) negotiation.cc:107] Waiting for socket to connect
0924 21:08:34.192395 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0924 21:08:34.192426 (+ 31us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 21:08:34.192532 (+ 106us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 21:08:34.192533 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 21:08:34.192640 (+ 107us) client_negotiation.cc:189] Negotiated authn=SASL
0924 21:08:34.192659 (+ 19us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:34.192660 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:34.193158 (+ 498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:34.193159 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:34.193593 (+ 434us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:34.193594 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:34.193977 (+ 383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:34.193978 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:34.194024 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:34.194355 (+ 331us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 21:08:34.194359 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 21:08:34.195658 (+ 1299us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34181: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":21}
W20260924 21:08:34.195761 9242 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 21:08:34.192392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:38367 (local address 127.0.98.62:34181)
0924 21:08:34.192420 (+ 28us) server_negotiation.cc:206] Beginning negotiation
0924 21:08:34.192421 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 21:08:34.192428 (+ 7us) server_negotiation.cc:407] Connection header received
0924 21:08:34.192475 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:08:34.192476 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:08:34.192483 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:08:34.192529 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:08:34.192718 (+ 189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:34.193090 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:34.193676 (+ 586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:34.193900 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:34.193976 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:34.195711 (+ 1735us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:38367: BlockingRecv error: recv got EOF from 127.0.98.1:38367 (error 108)
Metrics: {"server-negotiator.queue_time_us":14}
W20260924 21:08:34.195808 9239 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260924 21:08:34.258436 9243 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 21:08:34.258617 9243 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 21:08:34.258651 9243 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 21:08:34.259755 9243 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 21:08:34.259819 9243 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 21:08:34.259863 9243 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 21:08:34.261251 9243 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41267
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790284076800039-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790284076800039-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790284076800039-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:40221
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790284076800039-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=43493
--tserver_master_addrs=127.0.98.62:34181
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9243
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790284076800039-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 19:49:06 UTC on ccycloud.quasar-vhqjoc.root.comops.site
build id 4670864
I20260924 21:08:34.261492 9243 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 21:08:34.261688 9243 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 21:08:34.266187 9248 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 21:08:34.266171 9251 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 21:08:34.266320 9243 server_base.cc:1034] running on GCE node
W20260924 21:08:34.266202 9249 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 21:08:34.266563 9243 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 21:08:34.266722 9243 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 21:08:34.267848 9243 hybrid_clock.cc:648] HybridClock initialized: now 1790284114267836 us; error 25 us; skew 500 ppm
Sep 24 21:08:34 dist-test-slave-c7tk krb5kdc[8508](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284114, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 21:08:34.270018 9243 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260924 21:08:34.270440 9243 webserver.cc:460] Webserver started at http://127.0.98.2:43493/ using document root <none> and password file <none>
I20260924 21:08:34.270658 9243 fs_manager.cc:359] Metadata directory not provided
I20260924 21:08:34.270709 9243 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 21:08:34.271802 9243 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.001s sys 0.000s
I20260924 21:08:34.272287 9258 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 21:08:34.272433 9243 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 21:08:34.272516 9243 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790284076800039-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790284076800039-392-0/minicluster-data/ts-1/wal
uuid: "c4ad82d1f8b84c1fa92afa6a1c44839f"
format_stamp: "Formatted at 2026-09-24 21:08:33 on dist-test-slave-c7tk"
server_key: "d258f626c31ac7bec214578f350dd607"
server_key_iv: "e7ec44292d0b5d6bf6755aa5401ee03a"
server_key_version: "encryptionkey@0"
I20260924 21:08:34.272779 9243 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790284076800039-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790284076800039-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790284076800039-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 21:08:34.282362 9243 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 21:08:34.283816 9243 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 21:08:34.283946 9243 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 21:08:34.284116 9243 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 21:08:34.284353 9243 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 21:08:34.284397 9243 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 21:08:34.284425 9243 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 21:08:34.284447 9243 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 21:08:34.290305 9243 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:40221
I20260924 21:08:34.290352 9371 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:40221 every 8 connection(s)
I20260924 21:08:34.290637 9243 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790284076800039-392-0/minicluster-data/ts-1/data/info.pb
Sep 24 21:08:34 dist-test-slave-c7tk krb5kdc[8508](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 21:08:34 dist-test-slave-c7tk krb5kdc[8508](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260924 21:08:34.296764 9242 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 21:08:34.291218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:39043 (local address 127.0.98.62:34181)
0924 21:08:34.291273 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0924 21:08:34.291276 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 21:08:34.291470 (+ 194us) server_negotiation.cc:407] Connection header received
0924 21:08:34.291804 (+ 334us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:08:34.291805 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:08:34.291818 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:08:34.291872 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:08:34.292557 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:34.292965 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:34.293853 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:34.294097 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:34.294155 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:34.296712 (+ 2557us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:39043: BlockingRecv error: recv got EOF from 127.0.98.2:39043 (error 108)
Metrics: {"server-negotiator.queue_time_us":30}
I20260924 21:08:34.296804 9374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:34.291221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34181 (local address 127.0.98.2:39043)
0924 21:08:34.291351 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0924 21:08:34.291378 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0924 21:08:34.291723 (+ 345us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 21:08:34.291911 (+ 188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 21:08:34.291914 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 21:08:34.292347 (+ 433us) client_negotiation.cc:189] Negotiated authn=SASL
0924 21:08:34.292490 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:34.292493 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:34.293035 (+ 542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:34.293036 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:34.293712 (+ 676us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:34.293714 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:34.294169 (+ 455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:34.294169 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:34.294258 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:34.294643 (+ 385us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 21:08:34.294650 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0924 21:08:34.296638 (+ 1988us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34181: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":82,"threads_started":1}
W20260924 21:08:34.297097 9372 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:34181 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:34181: Client connection negotiation failed: client connection to 127.0.98.62:34181: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
I20260924 21:08:34.299872 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAzNNjS/build/release/bin/kudu as pid 9243
Sep 24 21:08:34 dist-test-slave-c7tk krb5kdc[8508](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 21:08:34.300208 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAzNNjS/build/release/bin/kudu
/tmp/dist-test-taskAzNNjS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790284076800039-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790284076800039-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790284076800039-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790284076800039-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:44627
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:34181
--webserver_port=43653
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790284076800039-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41267
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.3
--keytab_file=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790284076800039-392-0/krb5kdc/oryx_127.0.98.3.keytab with env {KRB5CCNAME=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790284076800039-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790284076800039-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790284076800039-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 21:08:34 dist-test-slave-c7tk krb5kdc[8508](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 21:08:34.300911 9374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:34.297307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34181 (local address 127.0.98.2:51745)
0924 21:08:34.297361 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0924 21:08:34.297371 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 21:08:34.297409 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 21:08:34.297506 (+ 97us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 21:08:34.297506 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 21:08:34.297594 (+ 88us) client_negotiation.cc:189] Negotiated authn=SASL
0924 21:08:34.297630 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:34.297630 (+ 0us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:34.298187 (+ 557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:34.298188 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:34.298576 (+ 388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:34.298577 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:34.298903 (+ 326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:34.298903 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:34.298943 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:34.299246 (+ 303us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 21:08:34.299249 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0924 21:08:34.300879 (+ 1630us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34181: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":39}
W20260924 21:08:34.301046 9242 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 21:08:34.297363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:51745 (local address 127.0.98.62:34181)
0924 21:08:34.297419 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0924 21:08:34.297421 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 21:08:34.297427 (+ 6us) server_negotiation.cc:407] Connection header received
0924 21:08:34.297448 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:08:34.297448 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:08:34.297455 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:08:34.297503 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:08:34.297679 (+ 176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:34.298105 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:34.298644 (+ 539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:34.298793 (+ 149us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:34.298884 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:34.300977 (+ 2093us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:51745: BlockingRecv error: recv got EOF from 127.0.98.2:51745 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
Sep 24 21:08:34 dist-test-slave-c7tk krb5kdc[8508](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 21:08:34 dist-test-slave-c7tk krb5kdc[8508](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 21:08:34.305392 9374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:34.301751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34181 (local address 127.0.98.2:36517)
0924 21:08:34.301787 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0924 21:08:34.301813 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0924 21:08:34.301845 (+ 32us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 21:08:34.301973 (+ 128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 21:08:34.301974 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 21:08:34.302091 (+ 117us) client_negotiation.cc:189] Negotiated authn=SASL
0924 21:08:34.302112 (+ 21us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:34.302113 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:34.302622 (+ 509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:34.302624 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:34.303078 (+ 454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:34.303080 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:34.303418 (+ 338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:34.303418 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:34.303465 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:34.303841 (+ 376us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 21:08:34.303844 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0924 21:08:34.305311 (+ 1467us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34181: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":26}
W20260924 21:08:34.305411 9242 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 21:08:34.301840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:36517 (local address 127.0.98.62:34181)
0924 21:08:34.301872 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0924 21:08:34.301874 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 21:08:34.301881 (+ 7us) server_negotiation.cc:407] Connection header received
0924 21:08:34.301898 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:08:34.301898 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:08:34.301905 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:08:34.301972 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:08:34.302182 (+ 210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:34.302549 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:34.303173 (+ 624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:34.303338 (+ 165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:34.303417 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:34.305378 (+ 1961us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:36517: BlockingRecv error: recv got EOF from 127.0.98.2:36517 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260924 21:08:34.305523 9372 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260924 21:08:34.371966 9375 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 21:08:34.372164 9375 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 21:08:34.372196 9375 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 21:08:34.373278 9375 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 21:08:34.373315 9375 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 21:08:34.373338 9375 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 21:08:34.374825 9375 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41267
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790284076800039-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790284076800039-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790284076800039-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:44627
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790284076800039-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=43653
--tserver_master_addrs=127.0.98.62:34181
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9375
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790284076800039-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 19:49:06 UTC on ccycloud.quasar-vhqjoc.root.comops.site
build id 4670864
I20260924 21:08:34.375027 9375 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 21:08:34.375231 9375 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 21:08:34.379662 9375 server_base.cc:1034] running on GCE node
W20260924 21:08:34.379784 9380 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 21:08:34.379685 9383 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 21:08:34.379701 9381 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 21:08:34.380113 9375 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 21:08:34.380357 9375 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 21:08:34.381491 9375 hybrid_clock.cc:648] HybridClock initialized: now 1790284114381488 us; error 38 us; skew 500 ppm
Sep 24 21:08:34 dist-test-slave-c7tk krb5kdc[8508](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284114, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 21:08:34.383551 9375 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260924 21:08:34.383929 9375 webserver.cc:460] Webserver started at http://127.0.98.3:43653/ using document root <none> and password file <none>
I20260924 21:08:34.384131 9375 fs_manager.cc:359] Metadata directory not provided
I20260924 21:08:34.384202 9375 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 21:08:34.385293 9375 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 21:08:34.385746 9390 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 21:08:34.385933 9375 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 21:08:34.386019 9375 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790284076800039-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790284076800039-392-0/minicluster-data/ts-2/wal
uuid: "85b113ef14db4e9aad1e42eadced86a8"
format_stamp: "Formatted at 2026-09-24 21:08:33 on dist-test-slave-c7tk"
server_key: "6f1565d771fc777dc6917a75def85c34"
server_key_iv: "f20fc01f0b8b72f9acb12bb9888ca3b9"
server_key_version: "encryptionkey@0"
I20260924 21:08:34.386323 9375 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790284076800039-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790284076800039-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790284076800039-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 21:08:34.399070 9375 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 21:08:34.400631 9375 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 21:08:34.400789 9375 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 21:08:34.400977 9375 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 21:08:34.401224 9375 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 21:08:34.401271 9375 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 21:08:34.401305 9375 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 21:08:34.401330 9375 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 21:08:34.406450 9375 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:44627
I20260924 21:08:34.406494 9503 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:44627 every 8 connection(s)
I20260924 21:08:34.406744 9375 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790284076800039-392-0/minicluster-data/ts-2/data/info.pb
Sep 24 21:08:34 dist-test-slave-c7tk krb5kdc[8508](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 21:08:34 dist-test-slave-c7tk krb5kdc[8508](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260924 21:08:34.412864 9242 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 21:08:34.407253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:42435 (local address 127.0.98.62:34181)
0924 21:08:34.407310 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0924 21:08:34.407311 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 21:08:34.407477 (+ 166us) server_negotiation.cc:407] Connection header received
0924 21:08:34.407839 (+ 362us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:08:34.407840 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:08:34.407848 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:08:34.407890 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:08:34.408645 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:34.409067 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:34.409997 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:34.410173 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:34.410225 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:34.412801 (+ 2576us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:42435: BlockingRecv error: recv got EOF from 127.0.98.3:42435 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
I20260924 21:08:34.412905 9506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:34.407245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34181 (local address 127.0.98.3:42435)
0924 21:08:34.407376 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0924 21:08:34.407403 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0924 21:08:34.407751 (+ 348us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 21:08:34.407938 (+ 187us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 21:08:34.407941 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 21:08:34.408418 (+ 477us) client_negotiation.cc:189] Negotiated authn=SASL
0924 21:08:34.408583 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:34.408586 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:34.409133 (+ 547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:34.409134 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:34.409933 (+ 799us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:34.409935 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:34.410239 (+ 304us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:34.410240 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:34.410326 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:34.410755 (+ 429us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 21:08:34.410762 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0924 21:08:34.412739 (+ 1977us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34181: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":80,"threads_started":1}
W20260924 21:08:34.413125 9504 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:34181 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:34181: Client connection negotiation failed: client connection to 127.0.98.62:34181: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
I20260924 21:08:34.414513 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAzNNjS/build/release/bin/kudu as pid 9375
Sep 24 21:08:34 dist-test-slave-c7tk krb5kdc[8508](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 21:08:34 dist-test-slave-c7tk krb5kdc[8508](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 21:08:34.417737 9506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:34.413404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34181 (local address 127.0.98.3:34177)
0924 21:08:34.413451 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0924 21:08:34.413461 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 21:08:34.413510 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 21:08:34.413754 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 21:08:34.413755 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 21:08:34.413909 (+ 154us) client_negotiation.cc:189] Negotiated authn=SASL
0924 21:08:34.413946 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:34.413947 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:34.414681 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:34.414682 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:34.415402 (+ 720us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:34.415404 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:34.415697 (+ 293us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:34.415698 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:34.415789 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:34.416151 (+ 362us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 21:08:34.416155 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 21:08:34.417676 (+ 1521us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34181: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":16}
W20260924 21:08:34.417874 9242 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 21:08:34.413606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:34177 (local address 127.0.98.62:34181)
0924 21:08:34.413655 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0924 21:08:34.413657 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 21:08:34.413663 (+ 6us) server_negotiation.cc:407] Connection header received
0924 21:08:34.413679 (+ 16us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:08:34.413679 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:08:34.413686 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:08:34.413750 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:08:34.414030 (+ 280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:34.414579 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:34.415479 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:34.415636 (+ 157us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:34.415678 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:34.417812 (+ 2134us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:34177: BlockingRecv error: recv got EOF from 127.0.98.3:34177 (error 108)
Metrics: {"server-negotiator.queue_time_us":32}
Sep 24 21:08:34 dist-test-slave-c7tk krb5kdc[8508](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 21:08:34 dist-test-slave-c7tk krb5kdc[8508](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 21:08:34.421648 9506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:34.417997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34181 (local address 127.0.98.3:52431)
0924 21:08:34.418039 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0924 21:08:34.418048 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 21:08:34.418084 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 21:08:34.418240 (+ 156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 21:08:34.418241 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 21:08:34.418362 (+ 121us) client_negotiation.cc:189] Negotiated authn=SASL
0924 21:08:34.418391 (+ 29us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:34.418392 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:34.418915 (+ 523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:34.418916 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:34.419363 (+ 447us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:34.419364 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:34.419810 (+ 446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:34.419811 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:34.419865 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:34.420158 (+ 293us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 21:08:34.420161 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0924 21:08:34.421585 (+ 1424us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34181: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":29}
W20260924 21:08:34.421690 9242 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 21:08:34.418052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:52431 (local address 127.0.98.62:34181)
0924 21:08:34.418080 (+ 28us) server_negotiation.cc:206] Beginning negotiation
0924 21:08:34.418081 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 21:08:34.418088 (+ 7us) server_negotiation.cc:407] Connection header received
0924 21:08:34.418179 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:08:34.418180 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:08:34.418189 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:08:34.418237 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:08:34.418456 (+ 219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:34.418847 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:34.419433 (+ 586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:34.419733 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:34.419811 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:34.421638 (+ 1827us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:52431: BlockingRecv error: recv got EOF from 127.0.98.3:52431 (error 108)
Metrics: {"server-negotiator.queue_time_us":14}
W20260924 21:08:34.421970 9504 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260924 21:08:34.422031 9508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:34.414880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37234 (local address 127.0.98.62:34181)
0924 21:08:34.414986 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0924 21:08:34.414988 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 21:08:34.415095 (+ 107us) server_negotiation.cc:407] Connection header received
0924 21:08:34.415152 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:08:34.415153 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:08:34.415161 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:08:34.415211 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:08:34.415929 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:34.416450 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:34.417384 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:34.417642 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:34.417700 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:34.418855 (+ 1155us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 21:08:34.418859 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 21:08:34.418859 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 21:08:34.418869 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0924 21:08:34.420727 (+ 1858us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:08:34.421084 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:08:34.421087 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:08:34.421088 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:08:34.421169 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:08:34.421369 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:08:34.421369 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:08:34.421370 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:08:34.421552 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 21:08:34.421681 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0924 21:08:34.421838 (+ 157us) server_negotiation.cc:299] Negotiation successful
0924 21:08:34.421934 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93,"thread_start_us":52,"threads_started":1}
Sep 24 21:08:35 dist-test-slave-c7tk krb5kdc[8508](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 21:08:35 dist-test-slave-c7tk krb5kdc[8508](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 21:08:35.200217 9509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:35.196162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34181 (local address 127.0.98.1:59035)
0924 21:08:35.196274 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0924 21:08:35.196302 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0924 21:08:35.196372 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 21:08:35.196491 (+ 119us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 21:08:35.196492 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 21:08:35.196714 (+ 222us) client_negotiation.cc:189] Negotiated authn=SASL
0924 21:08:35.196771 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:35.196773 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:35.197324 (+ 551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:35.197325 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:35.197831 (+ 506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:35.197834 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:35.198114 (+ 280us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:35.198115 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:35.198151 (+ 36us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:35.198594 (+ 443us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 21:08:35.198598 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 21:08:35.200157 (+ 1559us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34181: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":58,"threads_started":1}
W20260924 21:08:35.200305 9510 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 21:08:35.196194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:59035 (local address 127.0.98.62:34181)
0924 21:08:35.196282 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0924 21:08:35.196284 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 21:08:35.196381 (+ 97us) server_negotiation.cc:407] Connection header received
0924 21:08:35.196407 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:08:35.196408 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:08:35.196419 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:08:35.196473 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:08:35.196830 (+ 357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:35.197266 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:35.197896 (+ 630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:35.198060 (+ 164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:35.198104 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:35.200237 (+ 2133us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:59035: BlockingRecv error: recv got EOF from 127.0.98.1:59035 (error 108)
Metrics: {"server-negotiator.queue_time_us":70,"thread_start_us":39,"threads_started":1}
Sep 24 21:08:35 dist-test-slave-c7tk krb5kdc[8508](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 21:08:35 dist-test-slave-c7tk krb5kdc[8508](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 21:08:35.309720 9511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:35.306052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34181 (local address 127.0.98.2:33641)
0924 21:08:35.306168 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0924 21:08:35.306178 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 21:08:35.306231 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 21:08:35.306348 (+ 117us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 21:08:35.306349 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 21:08:35.306472 (+ 123us) client_negotiation.cc:189] Negotiated authn=SASL
0924 21:08:35.306511 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:35.306512 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:35.307048 (+ 536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:35.307048 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:35.307498 (+ 450us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:35.307499 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:35.307757 (+ 258us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:35.307758 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:35.307789 (+ 31us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:35.308207 (+ 418us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 21:08:35.308211 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 21:08:35.309629 (+ 1418us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34181: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":72,"threads_started":1}
W20260924 21:08:35.309785 9510 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 21:08:35.306137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:33641 (local address 127.0.98.62:34181)
0924 21:08:35.306196 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0924 21:08:35.306198 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 21:08:35.306221 (+ 23us) server_negotiation.cc:407] Connection header received
0924 21:08:35.306272 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:08:35.306272 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:08:35.306280 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:08:35.306325 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:08:35.306566 (+ 241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:35.306992 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:35.307559 (+ 567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:35.307705 (+ 146us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:35.307746 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:35.309741 (+ 1995us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:33641: BlockingRecv error: recv got EOF from 127.0.98.2:33641 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
Sep 24 21:08:35 dist-test-slave-c7tk krb5kdc[8508](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 21:08:35 dist-test-slave-c7tk krb5kdc[8508](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 21:08:35.426036 9512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:35.422257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34181 (local address 127.0.98.3:40485)
0924 21:08:35.422394 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0924 21:08:35.422404 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 21:08:35.422459 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 21:08:35.422564 (+ 105us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 21:08:35.422564 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 21:08:35.422693 (+ 129us) client_negotiation.cc:189] Negotiated authn=SASL
0924 21:08:35.422734 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:35.422735 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:35.423286 (+ 551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:35.423287 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:35.423758 (+ 471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:35.423760 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:35.424034 (+ 274us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:35.424034 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:35.424066 (+ 32us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:35.424493 (+ 427us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 21:08:35.424503 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 21:08:35.425979 (+ 1476us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34181: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":63,"threads_started":1}
W20260924 21:08:35.426095 9510 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 21:08:35.422348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:40485 (local address 127.0.98.62:34181)
0924 21:08:35.422385 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0924 21:08:35.422387 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 21:08:35.422458 (+ 71us) server_negotiation.cc:407] Connection header received
0924 21:08:35.422495 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:08:35.422495 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:08:35.422505 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:08:35.422545 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:08:35.422782 (+ 237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:35.423232 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:35.423824 (+ 592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:35.423985 (+ 161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:35.424025 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:35.426053 (+ 2028us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:40485: BlockingRecv error: recv got EOF from 127.0.98.3:40485 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
Sep 24 21:08:36 dist-test-slave-c7tk krb5kdc[8508](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 21:08:36 dist-test-slave-c7tk krb5kdc[8508](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 21:08:36.204481 9513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:36.200649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34181 (local address 127.0.98.1:41895)
0924 21:08:36.200765 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0924 21:08:36.200775 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 21:08:36.200829 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 21:08:36.200964 (+ 135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 21:08:36.200964 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 21:08:36.201119 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0924 21:08:36.201181 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:36.201182 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:36.201749 (+ 567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:36.201749 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:36.202272 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:36.202275 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:36.202579 (+ 304us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:36.202580 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:36.202614 (+ 34us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:36.202984 (+ 370us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 21:08:36.202988 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 21:08:36.204435 (+ 1447us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34181: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":69,"threads_started":1}
W20260924 21:08:36.204581 9514 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 21:08:36.200722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41895 (local address 127.0.98.62:34181)
0924 21:08:36.200843 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0924 21:08:36.200845 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 21:08:36.200854 (+ 9us) server_negotiation.cc:407] Connection header received
0924 21:08:36.200881 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:08:36.200882 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:08:36.200891 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:08:36.200959 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:08:36.201239 (+ 280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:36.201689 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:36.202356 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:36.202516 (+ 160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:36.202562 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:36.204513 (+ 1951us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:41895: BlockingRecv error: recv got EOF from 127.0.98.1:41895 (error 108)
Metrics: {"server-negotiator.queue_time_us":101,"thread_start_us":73,"threads_started":1}
Sep 24 21:08:36 dist-test-slave-c7tk krb5kdc[8508](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 21:08:36 dist-test-slave-c7tk krb5kdc[8508](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 21:08:36.313963 9515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:36.310189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34181 (local address 127.0.98.2:34781)
0924 21:08:36.310321 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0924 21:08:36.310332 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 21:08:36.310388 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 21:08:36.310499 (+ 111us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 21:08:36.310500 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 21:08:36.310657 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0924 21:08:36.310700 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:36.310701 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:36.311282 (+ 581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:36.311283 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:36.311743 (+ 460us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:36.311744 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:36.312001 (+ 257us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:36.312001 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:36.312034 (+ 33us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:36.312385 (+ 351us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 21:08:36.312389 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 21:08:36.313915 (+ 1526us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34181: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":80,"threads_started":1}
W20260924 21:08:36.314064 9514 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 21:08:36.310276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:34781 (local address 127.0.98.62:34181)
0924 21:08:36.310319 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0924 21:08:36.310321 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 21:08:36.310378 (+ 57us) server_negotiation.cc:407] Connection header received
0924 21:08:36.310445 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:08:36.310446 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:08:36.310455 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:08:36.310494 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:08:36.310757 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:36.311224 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:36.311798 (+ 574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:36.311947 (+ 149us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:36.311990 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:36.313982 (+ 1992us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:34781: BlockingRecv error: recv got EOF from 127.0.98.2:34781 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
Sep 24 21:08:36 dist-test-slave-c7tk krb5kdc[8508](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 21:08:36 dist-test-slave-c7tk krb5kdc[8508](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 21:08:36.429949 9516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:36.426444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34181 (local address 127.0.98.3:34459)
0924 21:08:36.426564 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0924 21:08:36.426573 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 21:08:36.426612 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 21:08:36.426727 (+ 115us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 21:08:36.426728 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 21:08:36.426856 (+ 128us) client_negotiation.cc:189] Negotiated authn=SASL
0924 21:08:36.426898 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:36.426900 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:36.427414 (+ 514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:36.427414 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:36.427841 (+ 427us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:36.427842 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:36.428092 (+ 250us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:36.428093 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:36.428125 (+ 32us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:36.428481 (+ 356us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 21:08:36.428485 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 21:08:36.429900 (+ 1415us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34181: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":90,"thread_start_us":52,"threads_started":1}
W20260924 21:08:36.430038 9514 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 21:08:36.426539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:34459 (local address 127.0.98.62:34181)
0924 21:08:36.426599 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0924 21:08:36.426600 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 21:08:36.426608 (+ 8us) server_negotiation.cc:407] Connection header received
0924 21:08:36.426661 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:08:36.426661 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:08:36.426670 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:08:36.426712 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:08:36.426950 (+ 238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:36.427360 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:36.427897 (+ 537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:36.428041 (+ 144us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:36.428081 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:36.429975 (+ 1894us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:34459: BlockingRecv error: recv got EOF from 127.0.98.3:34459 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
Sep 24 21:08:37 dist-test-slave-c7tk krb5kdc[8508](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 21:08:37 dist-test-slave-c7tk krb5kdc[8508](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 21:08:37.208899 9517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:37.204937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34181 (local address 127.0.98.1:37391)
0924 21:08:37.205072 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0924 21:08:37.205088 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 21:08:37.205128 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 21:08:37.205306 (+ 178us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 21:08:37.205307 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 21:08:37.205441 (+ 134us) client_negotiation.cc:189] Negotiated authn=SASL
0924 21:08:37.205485 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:37.205487 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:37.206046 (+ 559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:37.206047 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:37.206710 (+ 663us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:37.206711 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:37.206997 (+ 286us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:37.206997 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:37.207031 (+ 34us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:37.207408 (+ 377us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 21:08:37.207412 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 21:08:37.208847 (+ 1435us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34181: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":90,"threads_started":1}
W20260924 21:08:37.209012 9518 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 21:08:37.205065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:37391 (local address 127.0.98.62:34181)
0924 21:08:37.205200 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0924 21:08:37.205202 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 21:08:37.205209 (+ 7us) server_negotiation.cc:407] Connection header received
0924 21:08:37.205227 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:08:37.205227 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:08:37.205236 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:08:37.205291 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:08:37.205541 (+ 250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:37.205985 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:37.206770 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:37.206941 (+ 171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:37.206988 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:37.208921 (+ 1933us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:37391: BlockingRecv error: recv got EOF from 127.0.98.1:37391 (error 108)
Metrics: {"server-negotiator.queue_time_us":113,"thread_start_us":82,"threads_started":1}
Sep 24 21:08:37 dist-test-slave-c7tk krb5kdc[8508](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 21:08:37 dist-test-slave-c7tk krb5kdc[8508](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 21:08:37.318023 9519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:37.314349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34181 (local address 127.0.98.2:50491)
0924 21:08:37.314505 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0924 21:08:37.314514 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 21:08:37.314563 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 21:08:37.314659 (+ 96us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 21:08:37.314660 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 21:08:37.314795 (+ 135us) client_negotiation.cc:189] Negotiated authn=SASL
0924 21:08:37.314845 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:37.314846 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:37.315355 (+ 509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:37.315356 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:37.315786 (+ 430us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:37.315787 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:37.316051 (+ 264us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:37.316052 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:37.316083 (+ 31us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:37.316438 (+ 355us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 21:08:37.316442 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 21:08:37.317954 (+ 1512us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34181: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":96,"threads_started":1}
W20260924 21:08:37.318136 9518 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 21:08:37.314381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:50491 (local address 127.0.98.62:34181)
0924 21:08:37.314440 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0924 21:08:37.314442 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 21:08:37.314578 (+ 136us) server_negotiation.cc:407] Connection header received
0924 21:08:37.314602 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:08:37.314603 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:08:37.314611 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:08:37.314651 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:08:37.314894 (+ 243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:37.315300 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:37.315853 (+ 553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:37.316000 (+ 147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:37.316041 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:37.318068 (+ 2027us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:50491: BlockingRecv error: recv got EOF from 127.0.98.2:50491 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
Sep 24 21:08:37 dist-test-slave-c7tk krb5kdc[8508](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 21:08:37 dist-test-slave-c7tk krb5kdc[8508](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 21:08:37.435072 9520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:37.430408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34181 (local address 127.0.98.3:59925)
0924 21:08:37.430614 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0924 21:08:37.430625 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 21:08:37.430684 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 21:08:37.430852 (+ 168us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 21:08:37.430853 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 21:08:37.431019 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0924 21:08:37.431075 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:37.431077 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:37.431969 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:37.431970 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:37.432527 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:37.432529 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:37.432800 (+ 271us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:37.432800 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:37.432834 (+ 34us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:37.433196 (+ 362us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 21:08:37.433200 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 21:08:37.435000 (+ 1800us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34181: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":46,"threads_started":1}
W20260924 21:08:37.435155 9518 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 21:08:37.430518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:59925 (local address 127.0.98.62:34181)
0924 21:08:37.430556 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0924 21:08:37.430558 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 21:08:37.430750 (+ 192us) server_negotiation.cc:407] Connection header received
0924 21:08:37.430774 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:08:37.430775 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:08:37.430788 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:08:37.430837 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:08:37.431183 (+ 346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:37.431882 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:37.432603 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:37.432752 (+ 149us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:37.432795 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:37.435095 (+ 2300us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:59925: BlockingRecv error: recv got EOF from 127.0.98.3:59925 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
Sep 24 21:08:38 dist-test-slave-c7tk krb5kdc[8508](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 21:08:38 dist-test-slave-c7tk krb5kdc[8508](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 21:08:38.213361 9521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:38.209257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34181 (local address 127.0.98.1:53083)
0924 21:08:38.209383 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0924 21:08:38.209394 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 21:08:38.209446 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 21:08:38.209562 (+ 116us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 21:08:38.209563 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 21:08:38.209695 (+ 132us) client_negotiation.cc:189] Negotiated authn=SASL
0924 21:08:38.209737 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:38.209739 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:38.210387 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:38.210388 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:38.210903 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:38.210904 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:38.211232 (+ 328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:38.211232 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:38.211265 (+ 33us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:38.211656 (+ 391us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 21:08:38.211660 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 21:08:38.213276 (+ 1616us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34181: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":59,"threads_started":1}
W20260924 21:08:38.213399 9522 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 21:08:38.209308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:53083 (local address 127.0.98.62:34181)
0924 21:08:38.209426 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0924 21:08:38.209428 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 21:08:38.209434 (+ 6us) server_negotiation.cc:407] Connection header received
0924 21:08:38.209489 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:08:38.209490 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:08:38.209498 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:08:38.209546 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:08:38.209783 (+ 237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:38.210325 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:38.210965 (+ 640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:38.211160 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:38.211217 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:38.213340 (+ 2123us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:53083: BlockingRecv error: recv got EOF from 127.0.98.1:53083 (error 108)
Metrics: {"server-negotiator.queue_time_us":102,"thread_start_us":72,"threads_started":1}
Sep 24 21:08:38 dist-test-slave-c7tk krb5kdc[8508](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 21:08:38 dist-test-slave-c7tk krb5kdc[8508](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 21:08:38.322548 9523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:38.318401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34181 (local address 127.0.98.2:34997)
0924 21:08:38.318510 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0924 21:08:38.318519 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 21:08:38.318580 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 21:08:38.318694 (+ 114us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 21:08:38.318695 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 21:08:38.318849 (+ 154us) client_negotiation.cc:189] Negotiated authn=SASL
0924 21:08:38.318891 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:38.318893 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:38.319506 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:38.319507 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:38.320014 (+ 507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:38.320016 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:38.320386 (+ 370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:38.320387 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:38.320450 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:38.320877 (+ 427us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 21:08:38.320881 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 21:08:38.322463 (+ 1582us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34181: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":89,"thread_start_us":62,"threads_started":1}
W20260924 21:08:38.322594 9522 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 21:08:38.318524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:34997 (local address 127.0.98.62:34181)
0924 21:08:38.318584 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0924 21:08:38.318586 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 21:08:38.318593 (+ 7us) server_negotiation.cc:407] Connection header received
0924 21:08:38.318633 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:08:38.318634 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:08:38.318644 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:08:38.318682 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:08:38.318961 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:38.319420 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:38.320104 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:38.320313 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:38.320388 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:38.322536 (+ 2148us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:34997: BlockingRecv error: recv got EOF from 127.0.98.2:34997 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
Sep 24 21:08:38 dist-test-slave-c7tk krb5kdc[8508](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 21:08:38 dist-test-slave-c7tk krb5kdc[8508](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 21:08:38.439216 9524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:38.435463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34181 (local address 127.0.98.3:48655)
0924 21:08:38.435590 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0924 21:08:38.435599 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 21:08:38.435646 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 21:08:38.435746 (+ 100us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 21:08:38.435746 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 21:08:38.435893 (+ 147us) client_negotiation.cc:189] Negotiated authn=SASL
0924 21:08:38.435938 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:38.435939 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:38.436505 (+ 566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:38.436506 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:38.436963 (+ 457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:38.436965 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:38.437243 (+ 278us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:38.437244 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:38.437276 (+ 32us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:38.437629 (+ 353us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 21:08:38.437633 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 21:08:38.439169 (+ 1536us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34181: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":74,"threads_started":1}
W20260924 21:08:38.439337 9522 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 21:08:38.435527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:48655 (local address 127.0.98.62:34181)
0924 21:08:38.435568 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0924 21:08:38.435570 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 21:08:38.435634 (+ 64us) server_negotiation.cc:407] Connection header received
0924 21:08:38.435694 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:08:38.435695 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:08:38.435704 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:08:38.435740 (+ 36us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:08:38.435992 (+ 252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:38.436446 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:38.437033 (+ 587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:38.437185 (+ 152us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:38.437233 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:38.439242 (+ 2009us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:48655: BlockingRecv error: recv got EOF from 127.0.98.3:48655 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
Sep 24 21:08:39 dist-test-slave-c7tk krb5kdc[8508](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 21:08:39 dist-test-slave-c7tk krb5kdc[8508](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 21:08:39.222543 9525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:39.213867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34181 (local address 127.0.98.1:50335)
0924 21:08:39.213980 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0924 21:08:39.213991 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 21:08:39.214049 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 21:08:39.214221 (+ 172us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 21:08:39.214222 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 21:08:39.214368 (+ 146us) client_negotiation.cc:189] Negotiated authn=SASL
0924 21:08:39.214413 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:39.214414 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:39.214971 (+ 557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:39.214971 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:39.215420 (+ 449us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:39.215421 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:39.215703 (+ 282us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:39.215704 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:39.215737 (+ 33us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:39.220821 (+ 5084us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 21:08:39.220826 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 21:08:39.222482 (+ 1656us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34181: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":93,"thread_start_us":57,"threads_started":1}
W20260924 21:08:39.222640 9526 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 21:08:39.213911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:50335 (local address 127.0.98.62:34181)
0924 21:08:39.214026 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0924 21:08:39.214028 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 21:08:39.214056 (+ 28us) server_negotiation.cc:407] Connection header received
0924 21:08:39.214132 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:08:39.214133 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:08:39.214144 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:08:39.214202 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:08:39.214472 (+ 270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:39.214915 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:39.215477 (+ 562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:39.215650 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:39.215692 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:39.222571 (+ 6879us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:50335: BlockingRecv error: recv got EOF from 127.0.98.1:50335 (error 108)
Metrics: {"server-negotiator.queue_time_us":95,"thread_start_us":56,"threads_started":1}
Sep 24 21:08:39 dist-test-slave-c7tk krb5kdc[8508](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 21:08:39 dist-test-slave-c7tk krb5kdc[8508](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 21:08:39.326592 9527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:39.322938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34181 (local address 127.0.98.2:41057)
0924 21:08:39.323069 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0924 21:08:39.323078 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 21:08:39.323124 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 21:08:39.323227 (+ 103us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 21:08:39.323228 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 21:08:39.323364 (+ 136us) client_negotiation.cc:189] Negotiated authn=SASL
0924 21:08:39.323406 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:39.323407 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:39.323942 (+ 535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:39.323943 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:39.324373 (+ 430us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:39.324374 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:39.324638 (+ 264us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:39.324639 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:39.324669 (+ 30us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:39.325013 (+ 344us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 21:08:39.325017 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 21:08:39.326542 (+ 1525us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34181: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":50,"threads_started":1}
W20260924 21:08:39.326681 9526 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 21:08:39.323078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:41057 (local address 127.0.98.62:34181)
0924 21:08:39.323131 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0924 21:08:39.323133 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 21:08:39.323142 (+ 9us) server_negotiation.cc:407] Connection header received
0924 21:08:39.323164 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:08:39.323165 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:08:39.323175 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:08:39.323223 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:08:39.323460 (+ 237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:39.323885 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:39.324440 (+ 555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:39.324586 (+ 146us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:39.324628 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:39.326621 (+ 1993us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:41057: BlockingRecv error: recv got EOF from 127.0.98.2:41057 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
Sep 24 21:08:39 dist-test-slave-c7tk krb5kdc[8508](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 21:08:39 dist-test-slave-c7tk krb5kdc[8508](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 21:08:39.443902 9528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:39.439580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34181 (local address 127.0.98.3:55835)
0924 21:08:39.439767 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0924 21:08:39.439784 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 21:08:39.439876 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 21:08:39.440027 (+ 151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 21:08:39.440028 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 21:08:39.440237 (+ 209us) client_negotiation.cc:189] Negotiated authn=SASL
0924 21:08:39.440285 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:39.440286 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:39.440980 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:39.440982 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:39.441525 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:39.441526 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:39.441895 (+ 369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:39.441896 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:39.441951 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:39.442320 (+ 369us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 21:08:39.442324 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 21:08:39.443816 (+ 1492us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34181: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":97,"threads_started":1}
W20260924 21:08:39.443954 9526 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 21:08:39.439674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:55835 (local address 127.0.98.62:34181)
0924 21:08:39.439739 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0924 21:08:39.439741 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 21:08:39.439857 (+ 116us) server_negotiation.cc:407] Connection header received
0924 21:08:39.439952 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:08:39.439953 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:08:39.439967 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:08:39.440019 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:08:39.440381 (+ 362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:39.440892 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:39.441618 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:39.441790 (+ 172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:39.441895 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:39.443895 (+ 2000us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:55835: BlockingRecv error: recv got EOF from 127.0.98.3:55835 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
Sep 24 21:08:40 dist-test-slave-c7tk krb5kdc[8508](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 21:08:40 dist-test-slave-c7tk krb5kdc[8508](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 21:08:40.227020 9529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:40.222977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34181 (local address 127.0.98.1:55451)
0924 21:08:40.223114 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0924 21:08:40.223125 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 21:08:40.223187 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 21:08:40.223324 (+ 137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 21:08:40.223324 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 21:08:40.223476 (+ 152us) client_negotiation.cc:189] Negotiated authn=SASL
0924 21:08:40.223519 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:40.223520 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:40.224095 (+ 575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:40.224096 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:40.224595 (+ 499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:40.224596 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:40.224932 (+ 336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:40.224933 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:40.224985 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:40.225410 (+ 425us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 21:08:40.225414 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 21:08:40.226939 (+ 1525us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34181: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":90,"threads_started":1}
W20260924 21:08:40.227070 9530 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 21:08:40.223034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:55451 (local address 127.0.98.62:34181)
0924 21:08:40.223126 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0924 21:08:40.223127 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 21:08:40.223169 (+ 42us) server_negotiation.cc:407] Connection header received
0924 21:08:40.223233 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:08:40.223234 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:08:40.223246 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:08:40.223313 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:08:40.223585 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:40.224021 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:40.224671 (+ 650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:40.224853 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:40.224930 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:40.227009 (+ 2079us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:55451: BlockingRecv error: recv got EOF from 127.0.98.1:55451 (error 108)
Metrics: {"server-negotiator.queue_time_us":74,"thread_start_us":41,"threads_started":1}
Sep 24 21:08:40 dist-test-slave-c7tk krb5kdc[8508](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 21:08:40 dist-test-slave-c7tk krb5kdc[8508](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 21:08:40.331784 9531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:40.327244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34181 (local address 127.0.98.2:53869)
0924 21:08:40.327481 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0924 21:08:40.327516 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0924 21:08:40.327584 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 21:08:40.327751 (+ 167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 21:08:40.327751 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 21:08:40.327968 (+ 217us) client_negotiation.cc:189] Negotiated authn=SASL
0924 21:08:40.328019 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:40.328022 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:40.328576 (+ 554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:40.328577 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:40.329161 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:40.329162 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:40.329566 (+ 404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:40.329567 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:40.329616 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:40.330011 (+ 395us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 21:08:40.330015 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 21:08:40.331723 (+ 1708us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34181: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":167,"threads_started":1}
W20260924 21:08:40.331889 9530 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 21:08:40.327351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:53869 (local address 127.0.98.62:34181)
0924 21:08:40.327415 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0924 21:08:40.327417 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 21:08:40.327584 (+ 167us) server_negotiation.cc:407] Connection header received
0924 21:08:40.327637 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:08:40.327638 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:08:40.327650 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:08:40.327727 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:08:40.328085 (+ 358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:40.328515 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:40.329226 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:40.329415 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:40.329468 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:40.331808 (+ 2340us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:53869: BlockingRecv error: recv got EOF from 127.0.98.2:53869 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
Sep 24 21:08:40 dist-test-slave-c7tk krb5kdc[8508](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 21:08:40 dist-test-slave-c7tk krb5kdc[8508](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 21:08:40.448400 9532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:40.444505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34181 (local address 127.0.98.3:36295)
0924 21:08:40.444670 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0924 21:08:40.444677 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0924 21:08:40.444747 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 21:08:40.444927 (+ 180us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 21:08:40.444930 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 21:08:40.445104 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0924 21:08:40.445146 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:40.445148 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:40.445686 (+ 538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:40.445687 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:40.446145 (+ 458us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:40.446146 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:40.446421 (+ 275us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:40.446422 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:40.446454 (+ 32us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:40.446878 (+ 424us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 21:08:40.446882 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 21:08:40.448343 (+ 1461us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34181: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":69,"threads_started":1}
W20260924 21:08:40.448457 9530 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 21:08:40.444572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:36295 (local address 127.0.98.62:34181)
0924 21:08:40.444614 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0924 21:08:40.444616 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 21:08:40.444734 (+ 118us) server_negotiation.cc:407] Connection header received
0924 21:08:40.444840 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:08:40.444841 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:08:40.444854 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:08:40.444902 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:08:40.445206 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:40.445629 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:40.446206 (+ 577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:40.446366 (+ 160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:40.446410 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:40.448417 (+ 2007us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:36295: BlockingRecv error: recv got EOF from 127.0.98.3:36295 (error 108)
Metrics: {"server-negotiator.queue_time_us":14}
Sep 24 21:08:41 dist-test-slave-c7tk krb5kdc[8508](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 21:08:41 dist-test-slave-c7tk krb5kdc[8508](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 21:08:41.231341 9533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:41.227446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34181 (local address 127.0.98.1:53849)
0924 21:08:41.227573 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0924 21:08:41.227582 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 21:08:41.227624 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 21:08:41.227747 (+ 123us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 21:08:41.227747 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 21:08:41.227890 (+ 143us) client_negotiation.cc:189] Negotiated authn=SASL
0924 21:08:41.227935 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:41.227936 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:41.228463 (+ 527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:41.228463 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:41.228904 (+ 441us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:41.228905 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:41.229166 (+ 261us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:41.229167 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:41.229201 (+ 34us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:41.229562 (+ 361us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 21:08:41.229566 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 21:08:41.231284 (+ 1718us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34181: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":76,"threads_started":1}
W20260924 21:08:41.231484 9534 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 21:08:41.227501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:53849 (local address 127.0.98.62:34181)
0924 21:08:41.227619 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0924 21:08:41.227621 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 21:08:41.227630 (+ 9us) server_negotiation.cc:407] Connection header received
0924 21:08:41.227679 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:08:41.227680 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:08:41.227687 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:08:41.227731 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:08:41.227993 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:41.228411 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:41.228961 (+ 550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:41.229114 (+ 153us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:41.229156 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:41.231368 (+ 2212us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:53849: BlockingRecv error: recv got EOF from 127.0.98.1:53849 (error 108)
Metrics: {"server-negotiator.queue_time_us":101,"thread_start_us":73,"threads_started":1}
Sep 24 21:08:41 dist-test-slave-c7tk krb5kdc[8508](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 21:08:41 dist-test-slave-c7tk krb5kdc[8508](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 21:08:41.336436 9535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:41.332244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34181 (local address 127.0.98.2:38197)
0924 21:08:41.332386 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0924 21:08:41.332396 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 21:08:41.332447 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 21:08:41.332640 (+ 193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 21:08:41.332641 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 21:08:41.332823 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0924 21:08:41.332870 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:41.332872 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:41.333553 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:41.333554 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:41.334127 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:41.334129 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:41.334437 (+ 308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:41.334437 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:41.334475 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:41.334887 (+ 412us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 21:08:41.334891 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 21:08:41.336382 (+ 1491us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34181: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":73,"threads_started":1}
W20260924 21:08:41.336551 9534 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 21:08:41.332270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:38197 (local address 127.0.98.62:34181)
0924 21:08:41.332323 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0924 21:08:41.332325 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 21:08:41.332456 (+ 131us) server_negotiation.cc:407] Connection header received
0924 21:08:41.332545 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:08:41.332546 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:08:41.332559 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:08:41.332617 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:08:41.332934 (+ 317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:41.333475 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:41.334194 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:41.334376 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:41.334427 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:41.336461 (+ 2034us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:38197: BlockingRecv error: recv got EOF from 127.0.98.2:38197 (error 108)
Metrics: {"server-negotiator.queue_time_us":28}
Sep 24 21:08:41 dist-test-slave-c7tk krb5kdc[8508](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 21:08:41 dist-test-slave-c7tk krb5kdc[8508](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 21:08:41.453086 9536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:41.448923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34181 (local address 127.0.98.3:58117)
0924 21:08:41.449069 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0924 21:08:41.449081 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 21:08:41.449162 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 21:08:41.449318 (+ 156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 21:08:41.449319 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 21:08:41.449541 (+ 222us) client_negotiation.cc:189] Negotiated authn=SASL
0924 21:08:41.449593 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:41.449595 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:41.450228 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:41.450229 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:41.450744 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:41.450745 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:41.451053 (+ 308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:41.451053 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:41.451090 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:41.451515 (+ 425us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 21:08:41.451519 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 21:08:41.453024 (+ 1505us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34181: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":52,"threads_started":1}
W20260924 21:08:41.453181 9534 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 21:08:41.449113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:58117 (local address 127.0.98.62:34181)
0924 21:08:41.449165 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0924 21:08:41.449167 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 21:08:41.449210 (+ 43us) server_negotiation.cc:407] Connection header received
0924 21:08:41.449230 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:08:41.449231 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:08:41.449244 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:08:41.449289 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:08:41.449665 (+ 376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:41.450172 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:41.450826 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:41.450996 (+ 170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:41.451041 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:41.453113 (+ 2072us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:58117: BlockingRecv error: recv got EOF from 127.0.98.3:58117 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
Sep 24 21:08:42 dist-test-slave-c7tk krb5kdc[8508](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 21:08:42 dist-test-slave-c7tk krb5kdc[8508](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 21:08:42.236076 9537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:42.231902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34181 (local address 127.0.98.1:42225)
0924 21:08:42.232040 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0924 21:08:42.232051 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 21:08:42.232107 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 21:08:42.232317 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 21:08:42.232318 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 21:08:42.232507 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0924 21:08:42.232562 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:42.232565 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:42.233233 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:42.233234 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:42.233767 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:42.233769 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:42.234091 (+ 322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:42.234092 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:42.234127 (+ 35us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:42.234507 (+ 380us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 21:08:42.234511 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 21:08:42.236025 (+ 1514us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34181: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":62,"threads_started":1}
W20260924 21:08:42.236197 9538 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 21:08:42.232018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:42225 (local address 127.0.98.62:34181)
0924 21:08:42.232173 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0924 21:08:42.232175 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 21:08:42.232183 (+ 8us) server_negotiation.cc:407] Connection header received
0924 21:08:42.232219 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:08:42.232220 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:08:42.232232 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:08:42.232297 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:08:42.232631 (+ 334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:42.233141 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:42.233858 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:42.234038 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:42.234082 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:42.236101 (+ 2019us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:42225: BlockingRecv error: recv got EOF from 127.0.98.1:42225 (error 108)
Metrics: {"server-negotiator.queue_time_us":109,"thread_start_us":84,"threads_started":1}
Sep 24 21:08:42 dist-test-slave-c7tk krb5kdc[8508](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 21:08:42 dist-test-slave-c7tk krb5kdc[8508](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 21:08:42.341209 9539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:42.336839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34181 (local address 127.0.98.2:46713)
0924 21:08:42.336955 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0924 21:08:42.336966 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 21:08:42.337022 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 21:08:42.337133 (+ 111us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 21:08:42.337134 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 21:08:42.337286 (+ 152us) client_negotiation.cc:189] Negotiated authn=SASL
0924 21:08:42.337329 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:42.337331 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:42.338143 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:42.338144 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:42.338611 (+ 467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:42.338612 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:42.338914 (+ 302us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:42.338915 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:42.338990 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:42.339478 (+ 488us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 21:08:42.339483 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 21:08:42.341154 (+ 1671us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34181: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":69,"threads_started":1}
W20260924 21:08:42.341325 9538 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 21:08:42.336911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:46713 (local address 127.0.98.62:34181)
0924 21:08:42.336979 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0924 21:08:42.336980 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 21:08:42.337012 (+ 32us) server_negotiation.cc:407] Connection header received
0924 21:08:42.337060 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:08:42.337061 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:08:42.337069 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:08:42.337119 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:08:42.337387 (+ 268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:42.338078 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:42.338680 (+ 602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:42.338862 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:42.338927 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:42.341233 (+ 2306us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:46713: BlockingRecv error: recv got EOF from 127.0.98.2:46713 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
Sep 24 21:08:42 dist-test-slave-c7tk krb5kdc[8508](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 21:08:42 dist-test-slave-c7tk krb5kdc[8508](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 21:08:42.457509 9540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:42.453494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34181 (local address 127.0.98.3:45931)
0924 21:08:42.453592 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0924 21:08:42.453602 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 21:08:42.453669 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 21:08:42.453785 (+ 116us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 21:08:42.453785 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 21:08:42.454001 (+ 216us) client_negotiation.cc:189] Negotiated authn=SASL
0924 21:08:42.454054 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:42.454055 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:42.454650 (+ 595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:42.454650 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:42.455111 (+ 461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:42.455112 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:42.455381 (+ 269us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:42.455382 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:42.455433 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:42.455774 (+ 341us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 21:08:42.455777 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0924 21:08:42.457410 (+ 1633us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34181: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":78,"thread_start_us":39,"threads_started":1}
W20260924 21:08:42.457551 9538 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 21:08:42.453560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:45931 (local address 127.0.98.62:34181)
0924 21:08:42.453616 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0924 21:08:42.453618 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 21:08:42.453647 (+ 29us) server_negotiation.cc:407] Connection header received
0924 21:08:42.453717 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:08:42.453717 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:08:42.453729 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:08:42.453770 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:08:42.454110 (+ 340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:42.454592 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:42.455170 (+ 578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:42.455328 (+ 158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:42.455371 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:42.457475 (+ 2104us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:45931: BlockingRecv error: recv got EOF from 127.0.98.3:45931 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
Sep 24 21:08:43 dist-test-slave-c7tk krb5kdc[8508](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 21:08:43 dist-test-slave-c7tk krb5kdc[8508](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 21:08:43.240599 9541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:43.236420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34181 (local address 127.0.98.1:59411)
0924 21:08:43.236543 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0924 21:08:43.236552 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 21:08:43.236601 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 21:08:43.236711 (+ 110us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 21:08:43.236712 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 21:08:43.236856 (+ 144us) client_negotiation.cc:189] Negotiated authn=SASL
0924 21:08:43.236918 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:43.236919 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:43.237696 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:43.237697 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:43.238205 (+ 508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:43.238206 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:43.238473 (+ 267us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:43.238474 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:43.238507 (+ 33us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:43.238939 (+ 432us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 21:08:43.238944 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 21:08:43.240547 (+ 1603us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34181: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":70,"threads_started":1}
W20260924 21:08:43.240703 9542 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 21:08:43.236492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:59411 (local address 127.0.98.62:34181)
0924 21:08:43.236596 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0924 21:08:43.236597 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 21:08:43.236605 (+ 8us) server_negotiation.cc:407] Connection header received
0924 21:08:43.236639 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:08:43.236639 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:08:43.236647 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:08:43.236695 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:08:43.236972 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:43.237628 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:43.238259 (+ 631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:43.238420 (+ 161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:43.238464 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:43.240628 (+ 2164us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:59411: BlockingRecv error: recv got EOF from 127.0.98.1:59411 (error 108)
Metrics: {"server-negotiator.queue_time_us":83,"thread_start_us":62,"threads_started":1}
Sep 24 21:08:43 dist-test-slave-c7tk krb5kdc[8508](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 21:08:43 dist-test-slave-c7tk krb5kdc[8508](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 21:08:43.345875 9543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:43.341635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34181 (local address 127.0.98.2:45757)
0924 21:08:43.341835 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0924 21:08:43.341846 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 21:08:43.341926 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 21:08:43.342059 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 21:08:43.342060 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 21:08:43.342221 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0924 21:08:43.342261 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:43.342263 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:43.342924 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:43.342925 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:43.343402 (+ 477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:43.343403 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:43.343749 (+ 346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:43.343750 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:43.343808 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:43.344166 (+ 358us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 21:08:43.344170 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 21:08:43.345766 (+ 1596us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34181: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":122,"threads_started":1}
W20260924 21:08:43.345930 9542 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 21:08:43.341740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:45757 (local address 127.0.98.62:34181)
0924 21:08:43.341832 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0924 21:08:43.341834 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 21:08:43.341908 (+ 74us) server_negotiation.cc:407] Connection header received
0924 21:08:43.341981 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:08:43.341982 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:08:43.341993 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:08:43.342055 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:08:43.342352 (+ 297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:43.342852 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:43.343491 (+ 639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:43.343676 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:43.343741 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:43.345870 (+ 2129us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:45757: BlockingRecv error: recv got EOF from 127.0.98.2:45757 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Sep 24 21:08:43 dist-test-slave-c7tk krb5kdc[8508](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 21:08:43 dist-test-slave-c7tk krb5kdc[8508](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 21:08:43.462092 9544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:43.458102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34181 (local address 127.0.98.3:47925)
0924 21:08:43.458223 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0924 21:08:43.458231 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0924 21:08:43.458280 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 21:08:43.458386 (+ 106us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 21:08:43.458386 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 21:08:43.458515 (+ 129us) client_negotiation.cc:189] Negotiated authn=SASL
0924 21:08:43.458556 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:43.458558 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:43.459075 (+ 517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:43.459076 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:43.459504 (+ 428us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:43.459505 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:43.459766 (+ 261us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:43.459767 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:43.459807 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:43.460227 (+ 420us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 21:08:43.460231 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 21:08:43.462045 (+ 1814us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34181: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":67,"threads_started":1}
W20260924 21:08:43.462167 9542 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 21:08:43.458168 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:47925 (local address 127.0.98.62:34181)
0924 21:08:43.458224 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0924 21:08:43.458225 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 21:08:43.458279 (+ 54us) server_negotiation.cc:407] Connection header received
0924 21:08:43.458315 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:08:43.458316 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:08:43.458324 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:08:43.458368 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:08:43.458610 (+ 242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:43.459022 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:43.459567 (+ 545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:43.459710 (+ 143us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:43.459751 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:43.462109 (+ 2358us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:47925: BlockingRecv error: recv got EOF from 127.0.98.3:47925 (error 108)
Metrics: {"server-negotiator.queue_time_us":31}
Sep 24 21:08:44 dist-test-slave-c7tk krb5kdc[8508](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 21:08:44 dist-test-slave-c7tk krb5kdc[8508](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 21:08:44.250524 9545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:44.241029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34181 (local address 127.0.98.1:58789)
0924 21:08:44.241157 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0924 21:08:44.241170 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 21:08:44.241233 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 21:08:44.241352 (+ 119us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 21:08:44.241353 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 21:08:44.241498 (+ 145us) client_negotiation.cc:189] Negotiated authn=SASL
0924 21:08:44.241546 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:44.241547 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:44.242135 (+ 588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:44.242135 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:44.242568 (+ 433us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:44.242569 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:44.242837 (+ 268us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:44.242838 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:44.242872 (+ 34us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:44.248484 (+ 5612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 21:08:44.248489 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 21:08:44.250455 (+ 1966us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34181: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":62,"threads_started":1}
W20260924 21:08:44.250669 9546 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 21:08:44.241074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:58789 (local address 127.0.98.62:34181)
0924 21:08:44.241209 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0924 21:08:44.241211 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 21:08:44.241219 (+ 8us) server_negotiation.cc:407] Connection header received
0924 21:08:44.241275 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:08:44.241276 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:08:44.241284 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:08:44.241337 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:08:44.241602 (+ 265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:44.242083 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:44.242625 (+ 542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:44.242786 (+ 161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:44.242826 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:44.250543 (+ 7717us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:58789: BlockingRecv error: recv got EOF from 127.0.98.1:58789 (error 108)
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":73,"threads_started":1}
Sep 24 21:08:44 dist-test-slave-c7tk krb5kdc[8508](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 21:08:44 dist-test-slave-c7tk krb5kdc[8508](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 21:08:44.350495 9547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:44.346439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34181 (local address 127.0.98.2:43057)
0924 21:08:44.346565 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0924 21:08:44.346575 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 21:08:44.346657 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 21:08:44.346824 (+ 167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 21:08:44.346824 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 21:08:44.347029 (+ 205us) client_negotiation.cc:189] Negotiated authn=SASL
0924 21:08:44.347073 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:44.347075 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:44.347640 (+ 565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:44.347640 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:44.348103 (+ 463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:44.348104 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:44.348379 (+ 275us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:44.348380 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:44.348415 (+ 35us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:44.348830 (+ 415us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 21:08:44.348834 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 21:08:44.350445 (+ 1611us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34181: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":70,"threads_started":1}
W20260924 21:08:44.350603 9546 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 21:08:44.346503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:43057 (local address 127.0.98.62:34181)
0924 21:08:44.346548 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0924 21:08:44.346550 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 21:08:44.346641 (+ 91us) server_negotiation.cc:407] Connection header received
0924 21:08:44.346725 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:08:44.346726 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:08:44.346737 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:08:44.346796 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:08:44.347136 (+ 340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:44.347582 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:44.348163 (+ 581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:44.348325 (+ 162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:44.348369 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:44.350522 (+ 2153us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:43057: BlockingRecv error: recv got EOF from 127.0.98.2:43057 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
Sep 24 21:08:44 dist-test-slave-c7tk krb5kdc[8508](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 21:08:44 dist-test-slave-c7tk krb5kdc[8508](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 21:08:44.467058 9548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:44.462724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34181 (local address 127.0.98.3:49511)
0924 21:08:44.462882 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0924 21:08:44.462894 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 21:08:44.462944 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 21:08:44.463074 (+ 130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 21:08:44.463075 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 21:08:44.463213 (+ 138us) client_negotiation.cc:189] Negotiated authn=SASL
0924 21:08:44.463258 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:44.463259 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:44.464004 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:44.464005 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:44.464503 (+ 498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:44.464505 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:44.464863 (+ 358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:44.464864 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:44.464923 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:44.465350 (+ 427us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 21:08:44.465354 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 21:08:44.467007 (+ 1653us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34181: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":110,"threads_started":1}
W20260924 21:08:44.467151 9546 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 21:08:44.462835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:49511 (local address 127.0.98.62:34181)
0924 21:08:44.462880 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0924 21:08:44.462882 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 21:08:44.462954 (+ 72us) server_negotiation.cc:407] Connection header received
0924 21:08:44.462978 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:08:44.462978 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:08:44.463004 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:08:44.463063 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:08:44.463333 (+ 270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:44.463875 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:44.464586 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:44.464772 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:44.464863 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:44.467087 (+ 2224us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:49511: BlockingRecv error: recv got EOF from 127.0.98.3:49511 (error 108)
Metrics: {"server-negotiator.queue_time_us":26}
Sep 24 21:08:45 dist-test-slave-c7tk krb5kdc[8508](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 21:08:45 dist-test-slave-c7tk krb5kdc[8508](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 21:08:45.254920 9549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:45.250986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34181 (local address 127.0.98.1:36469)
0924 21:08:45.251120 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0924 21:08:45.251130 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 21:08:45.251174 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 21:08:45.251302 (+ 128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 21:08:45.251303 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 21:08:45.251447 (+ 144us) client_negotiation.cc:189] Negotiated authn=SASL
0924 21:08:45.251493 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:45.251494 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:45.252079 (+ 585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:45.252080 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:45.252518 (+ 438us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:45.252519 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:45.252793 (+ 274us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:45.252793 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:45.252849 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:45.253253 (+ 404us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 21:08:45.253257 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 21:08:45.254872 (+ 1615us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34181: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":77,"threads_started":1}
W20260924 21:08:45.255046 9550 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 21:08:45.251063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:36469 (local address 127.0.98.62:34181)
0924 21:08:45.251153 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0924 21:08:45.251155 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 21:08:45.251182 (+ 27us) server_negotiation.cc:407] Connection header received
0924 21:08:45.251224 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:08:45.251225 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:08:45.251234 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:08:45.251282 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:08:45.251552 (+ 270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:45.252026 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:45.252581 (+ 555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:45.252734 (+ 153us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:45.252775 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:45.254968 (+ 2193us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:36469: BlockingRecv error: recv got EOF from 127.0.98.1:36469 (error 108)
Metrics: {"server-negotiator.queue_time_us":71,"thread_start_us":46,"threads_started":1}
Sep 24 21:08:45 dist-test-slave-c7tk krb5kdc[8508](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 21:08:45 dist-test-slave-c7tk krb5kdc[8508](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 21:08:45.355577 9551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:45.351017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34181 (local address 127.0.98.2:53205)
0924 21:08:45.351191 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0924 21:08:45.351202 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 21:08:45.351290 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 21:08:45.351448 (+ 158us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 21:08:45.351448 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 21:08:45.351615 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0924 21:08:45.351659 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:45.351661 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:45.352404 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:45.352405 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:45.352917 (+ 512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:45.352919 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:45.353258 (+ 339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:45.353259 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:45.353311 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:45.353854 (+ 543us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 21:08:45.353859 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 21:08:45.355479 (+ 1620us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34181: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":89,"threads_started":1}
W20260924 21:08:45.355624 9550 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 21:08:45.351099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:53205 (local address 127.0.98.62:34181)
0924 21:08:45.351173 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0924 21:08:45.351175 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 21:08:45.351271 (+ 96us) server_negotiation.cc:407] Connection header received
0924 21:08:45.351355 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:08:45.351356 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:08:45.351368 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:08:45.351441 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:08:45.351744 (+ 303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:45.352330 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:45.352983 (+ 653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:45.353185 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:45.353257 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:45.355568 (+ 2311us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:53205: BlockingRecv error: recv got EOF from 127.0.98.2:53205 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Sep 24 21:08:45 dist-test-slave-c7tk krb5kdc[8508](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 21:08:45 dist-test-slave-c7tk krb5kdc[8508](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 21:08:45.471594 9552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:45.467473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34181 (local address 127.0.98.3:46267)
0924 21:08:45.467592 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0924 21:08:45.467600 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0924 21:08:45.467647 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 21:08:45.467767 (+ 120us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 21:08:45.467768 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 21:08:45.467914 (+ 146us) client_negotiation.cc:189] Negotiated authn=SASL
0924 21:08:45.467957 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:45.467958 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:45.468545 (+ 587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:45.468546 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:45.469008 (+ 462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:45.469009 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:45.469283 (+ 274us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:45.469284 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:45.469315 (+ 31us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:45.469745 (+ 430us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 21:08:45.469749 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 21:08:45.471506 (+ 1757us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34181: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":72,"threads_started":1}
W20260924 21:08:45.471637 9550 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 21:08:45.467508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:46267 (local address 127.0.98.62:34181)
0924 21:08:45.467562 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0924 21:08:45.467564 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 21:08:45.467650 (+ 86us) server_negotiation.cc:407] Connection header received
0924 21:08:45.467690 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:08:45.467691 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:08:45.467699 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:08:45.467734 (+ 35us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:08:45.468021 (+ 287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:45.468456 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:45.469087 (+ 631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:45.469232 (+ 145us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:45.469278 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:45.471590 (+ 2312us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:46267: BlockingRecv error: recv got EOF from 127.0.98.3:46267 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
Sep 24 21:08:46 dist-test-slave-c7tk krb5kdc[8508](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 21:08:46 dist-test-slave-c7tk krb5kdc[8508](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 21:08:46.259377 9553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:46.255374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34181 (local address 127.0.98.1:58371)
0924 21:08:46.255520 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0924 21:08:46.255529 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 21:08:46.255575 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 21:08:46.255699 (+ 124us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 21:08:46.255700 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 21:08:46.255843 (+ 143us) client_negotiation.cc:189] Negotiated authn=SASL
0924 21:08:46.255902 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:46.255904 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:46.256483 (+ 579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:46.256484 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:46.256937 (+ 453us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:46.256939 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:46.257261 (+ 322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:46.257262 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:46.257311 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:46.257735 (+ 424us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 21:08:46.257739 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 21:08:46.259318 (+ 1579us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34181: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":77,"threads_started":1}
W20260924 21:08:46.259464 9554 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 21:08:46.255441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:58371 (local address 127.0.98.62:34181)
0924 21:08:46.255550 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0924 21:08:46.255552 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 21:08:46.255564 (+ 12us) server_negotiation.cc:407] Connection header received
0924 21:08:46.255621 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:08:46.255622 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:08:46.255630 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:08:46.255685 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:08:46.255960 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:46.256424 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:46.257025 (+ 601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:46.257203 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:46.257251 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:46.259398 (+ 2147us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:58371: BlockingRecv error: recv got EOF from 127.0.98.1:58371 (error 108)
Metrics: {"server-negotiator.queue_time_us":88,"thread_start_us":41,"threads_started":1}
Sep 24 21:08:46 dist-test-slave-c7tk krb5kdc[8508](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 21:08:46 dist-test-slave-c7tk krb5kdc[8508](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 21:08:46.360147 9555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:46.355986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34181 (local address 127.0.98.2:36621)
0924 21:08:46.356099 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0924 21:08:46.356116 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 21:08:46.356169 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 21:08:46.356282 (+ 113us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 21:08:46.356283 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 21:08:46.356426 (+ 143us) client_negotiation.cc:189] Negotiated authn=SASL
0924 21:08:46.356471 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:46.356472 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:46.357097 (+ 625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:46.357098 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:46.357887 (+ 789us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:46.357890 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:46.358161 (+ 271us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:46.358161 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:46.358192 (+ 31us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:46.358608 (+ 416us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 21:08:46.358612 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 21:08:46.360084 (+ 1472us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34181: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":68,"threads_started":1}
W20260924 21:08:46.360211 9554 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 21:08:46.356057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:36621 (local address 127.0.98.62:34181)
0924 21:08:46.356111 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0924 21:08:46.356112 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 21:08:46.356173 (+ 61us) server_negotiation.cc:407] Connection header received
0924 21:08:46.356211 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:08:46.356211 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:08:46.356220 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:08:46.356274 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:08:46.356528 (+ 254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:46.357013 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:46.357947 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:46.358104 (+ 157us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:46.358153 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:46.360167 (+ 2014us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:36621: BlockingRecv error: recv got EOF from 127.0.98.2:36621 (error 108)
Metrics: {"server-negotiator.queue_time_us":32}
Sep 24 21:08:46 dist-test-slave-c7tk krb5kdc[8508](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 21:08:46 dist-test-slave-c7tk krb5kdc[8508](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 21:08:46.476016 9556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:46.472038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34181 (local address 127.0.98.3:46157)
0924 21:08:46.472158 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0924 21:08:46.472174 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 21:08:46.472218 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 21:08:46.472329 (+ 111us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 21:08:46.472330 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 21:08:46.472455 (+ 125us) client_negotiation.cc:189] Negotiated authn=SASL
0924 21:08:46.472499 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:46.472500 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:46.473013 (+ 513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:46.473013 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:46.473459 (+ 446us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:46.473460 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:46.473758 (+ 298us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:46.473758 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:46.473833 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:46.474388 (+ 555us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 21:08:46.474392 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 21:08:46.475937 (+ 1545us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34181: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":80,"threads_started":1}
W20260924 21:08:46.476051 9554 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 21:08:46.472108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:46157 (local address 127.0.98.62:34181)
0924 21:08:46.472150 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0924 21:08:46.472152 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 21:08:46.472206 (+ 54us) server_negotiation.cc:407] Connection header received
0924 21:08:46.472254 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:08:46.472254 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:08:46.472270 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:08:46.472314 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:08:46.472553 (+ 239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:46.472958 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:46.473536 (+ 578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:46.473689 (+ 153us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:46.473733 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:46.475998 (+ 2265us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:46157: BlockingRecv error: recv got EOF from 127.0.98.3:46157 (error 108)
Metrics: {"server-negotiator.queue_time_us":15}
Sep 24 21:08:47 dist-test-slave-c7tk krb5kdc[8508](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 21:08:47 dist-test-slave-c7tk krb5kdc[8508](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 21:08:47.264165 9557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:47.259997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34181 (local address 127.0.98.1:40677)
0924 21:08:47.260121 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0924 21:08:47.260131 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 21:08:47.260174 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 21:08:47.260300 (+ 126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 21:08:47.260301 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 21:08:47.260441 (+ 140us) client_negotiation.cc:189] Negotiated authn=SASL
0924 21:08:47.260485 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:47.260486 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:47.261104 (+ 618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:47.261105 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:47.261613 (+ 508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:47.261614 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:47.261989 (+ 375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:47.261990 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:47.262028 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:47.262437 (+ 409us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 21:08:47.262441 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 21:08:47.264084 (+ 1643us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34181: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":60,"threads_started":1}
W20260924 21:08:47.264204 9558 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 21:08:47.260050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:40677 (local address 127.0.98.62:34181)
0924 21:08:47.260193 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0924 21:08:47.260195 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 21:08:47.260203 (+ 8us) server_negotiation.cc:407] Connection header received
0924 21:08:47.260222 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:08:47.260223 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:08:47.260232 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:08:47.260291 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:08:47.260543 (+ 252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:47.261043 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:47.261679 (+ 636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:47.261919 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:47.261983 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:47.264144 (+ 2161us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:40677: BlockingRecv error: recv got EOF from 127.0.98.1:40677 (error 108)
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":88,"threads_started":1}
Sep 24 21:08:47 dist-test-slave-c7tk krb5kdc[8508](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 21:08:47 dist-test-slave-c7tk krb5kdc[8508](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 21:08:47.365458 9559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:47.360760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34181 (local address 127.0.98.2:56231)
0924 21:08:47.360933 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0924 21:08:47.360944 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 21:08:47.361006 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 21:08:47.361141 (+ 135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 21:08:47.361141 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 21:08:47.361293 (+ 152us) client_negotiation.cc:189] Negotiated authn=SASL
0924 21:08:47.361337 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:47.361339 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:47.361999 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:47.362000 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:47.362810 (+ 810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:47.362814 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:47.363272 (+ 458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:47.363273 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:47.363330 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:47.363728 (+ 398us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 21:08:47.363733 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 21:08:47.365370 (+ 1637us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34181: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":69,"threads_started":1}
W20260924 21:08:47.365496 9558 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 21:08:47.360890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:56231 (local address 127.0.98.62:34181)
0924 21:08:47.360968 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0924 21:08:47.360969 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 21:08:47.360983 (+ 14us) server_negotiation.cc:407] Connection header received
0924 21:08:47.361066 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:08:47.361066 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:08:47.361076 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:08:47.361124 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:08:47.361398 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:47.361937 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:47.362880 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:47.363180 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:47.363265 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:47.365443 (+ 2178us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:56231: BlockingRecv error: recv got EOF from 127.0.98.2:56231 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Sep 24 21:08:47 dist-test-slave-c7tk krb5kdc[8508](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 21:08:47 dist-test-slave-c7tk krb5kdc[8508](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 21:08:47.480347 9560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:47.476393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34181 (local address 127.0.98.3:41263)
0924 21:08:47.476491 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0924 21:08:47.476505 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 21:08:47.476552 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 21:08:47.476673 (+ 121us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 21:08:47.476674 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 21:08:47.476799 (+ 125us) client_negotiation.cc:189] Negotiated authn=SASL
0924 21:08:47.476841 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:47.476842 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:47.477421 (+ 579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:47.477421 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:47.477869 (+ 448us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:47.477871 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:47.478157 (+ 286us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:47.478157 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:47.478192 (+ 35us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:47.478639 (+ 447us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 21:08:47.478643 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 21:08:47.480286 (+ 1643us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34181: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":81,"thread_start_us":58,"threads_started":1}
W20260924 21:08:47.480396 9558 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 21:08:47.476456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:41263 (local address 127.0.98.62:34181)
0924 21:08:47.476506 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0924 21:08:47.476510 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 21:08:47.476539 (+ 29us) server_negotiation.cc:407] Connection header received
0924 21:08:47.476600 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:08:47.476601 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:08:47.476611 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:08:47.476661 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:08:47.476897 (+ 236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:47.477357 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:47.477927 (+ 570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:47.478079 (+ 152us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:47.478151 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:47.480354 (+ 2203us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:41263: BlockingRecv error: recv got EOF from 127.0.98.3:41263 (error 108)
Metrics: {"server-negotiator.queue_time_us":30}
Sep 24 21:08:48 dist-test-slave-c7tk krb5kdc[8508](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 21:08:48 dist-test-slave-c7tk krb5kdc[8508](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 21:08:48.268349 9561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:48.264559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34181 (local address 127.0.98.1:42839)
0924 21:08:48.264658 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0924 21:08:48.264669 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 21:08:48.264725 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 21:08:48.264839 (+ 114us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 21:08:48.264839 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 21:08:48.264972 (+ 133us) client_negotiation.cc:189] Negotiated authn=SASL
0924 21:08:48.265016 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:48.265018 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:48.265556 (+ 538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:48.265556 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:48.266052 (+ 496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:48.266054 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:48.266322 (+ 268us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:48.266323 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:48.266355 (+ 32us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:48.266699 (+ 344us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 21:08:48.266703 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 21:08:48.268264 (+ 1561us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34181: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":79,"thread_start_us":47,"threads_started":1}
W20260924 21:08:48.268406 9562 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 21:08:48.264610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:42839 (local address 127.0.98.62:34181)
0924 21:08:48.264721 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0924 21:08:48.264722 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 21:08:48.264729 (+ 7us) server_negotiation.cc:407] Connection header received
0924 21:08:48.264768 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:08:48.264769 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:08:48.264779 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:08:48.264821 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:08:48.265070 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:48.265502 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:48.266112 (+ 610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:48.266266 (+ 154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:48.266313 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:48.268341 (+ 2028us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:42839: BlockingRecv error: recv got EOF from 127.0.98.1:42839 (error 108)
Metrics: {"server-negotiator.queue_time_us":93,"thread_start_us":54,"threads_started":1}
Sep 24 21:08:48 dist-test-slave-c7tk krb5kdc[8508](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 21:08:48 dist-test-slave-c7tk krb5kdc[8508](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 21:08:48.369784 9563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:48.365889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34181 (local address 127.0.98.2:49831)
0924 21:08:48.365985 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0924 21:08:48.366001 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 21:08:48.366053 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 21:08:48.366165 (+ 112us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 21:08:48.366166 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 21:08:48.366304 (+ 138us) client_negotiation.cc:189] Negotiated authn=SASL
0924 21:08:48.366347 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:48.366349 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:48.366867 (+ 518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:48.366868 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:48.367301 (+ 433us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:48.367301 (+ 0us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:48.367555 (+ 254us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:48.367555 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:48.367587 (+ 32us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:48.367986 (+ 399us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 21:08:48.367990 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 21:08:48.369692 (+ 1702us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34181: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":78,"thread_start_us":46,"threads_started":1}
W20260924 21:08:48.369830 9562 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 21:08:48.365950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:49831 (local address 127.0.98.62:34181)
0924 21:08:48.366003 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0924 21:08:48.366004 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 21:08:48.366051 (+ 47us) server_negotiation.cc:407] Connection header received
0924 21:08:48.366092 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:08:48.366092 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:08:48.366100 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:08:48.366148 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:08:48.366398 (+ 250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:48.366812 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:48.367355 (+ 543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:48.367503 (+ 148us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:48.367544 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:48.369759 (+ 2215us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:49831: BlockingRecv error: recv got EOF from 127.0.98.2:49831 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
Sep 24 21:08:48 dist-test-slave-c7tk krb5kdc[8508](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 21:08:48 dist-test-slave-c7tk krb5kdc[8508](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 21:08:48.484948 9564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:48.480969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34181 (local address 127.0.98.3:45871)
0924 21:08:48.481108 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0924 21:08:48.481125 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 21:08:48.481179 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 21:08:48.481295 (+ 116us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 21:08:48.481296 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 21:08:48.481440 (+ 144us) client_negotiation.cc:189] Negotiated authn=SASL
0924 21:08:48.481484 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:48.481486 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:48.482102 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:48.482103 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:48.482581 (+ 478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:48.482582 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:48.482873 (+ 291us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:48.482874 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:48.482908 (+ 34us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:48.483335 (+ 427us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 21:08:48.483339 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 21:08:48.484897 (+ 1558us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34181: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":91,"threads_started":1}
W20260924 21:08:48.485059 9562 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 21:08:48.481072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:45871 (local address 127.0.98.62:34181)
0924 21:08:48.481133 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0924 21:08:48.481134 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 21:08:48.481179 (+ 45us) server_negotiation.cc:407] Connection header received
0924 21:08:48.481219 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:08:48.481219 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:08:48.481229 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:08:48.481283 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:08:48.481539 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:48.482039 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:48.482640 (+ 601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:48.482808 (+ 168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:48.482863 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:48.484984 (+ 2121us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:45871: BlockingRecv error: recv got EOF from 127.0.98.3:45871 (error 108)
Metrics: {"server-negotiator.queue_time_us":32}
Sep 24 21:08:49 dist-test-slave-c7tk krb5kdc[8508](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 21:08:49 dist-test-slave-c7tk krb5kdc[8508](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 21:08:49.278534 9565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:49.268865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34181 (local address 127.0.98.1:46881)
0924 21:08:49.269019 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0924 21:08:49.269029 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 21:08:49.269087 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 21:08:49.269235 (+ 148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 21:08:49.269236 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 21:08:49.269384 (+ 148us) client_negotiation.cc:189] Negotiated authn=SASL
0924 21:08:49.269430 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:49.269431 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:49.270140 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:49.270141 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:49.270641 (+ 500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:49.270643 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:49.270941 (+ 298us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:49.270941 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:49.270975 (+ 34us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:49.276807 (+ 5832us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 21:08:49.276811 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 21:08:49.278476 (+ 1665us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34181: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":70,"threads_started":1}
W20260924 21:08:49.278671 9566 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 21:08:49.268939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:46881 (local address 127.0.98.62:34181)
0924 21:08:49.269059 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0924 21:08:49.269062 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 21:08:49.269077 (+ 15us) server_negotiation.cc:407] Connection header received
0924 21:08:49.269131 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:08:49.269132 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:08:49.269141 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:08:49.269225 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:08:49.269497 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:49.270078 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:49.270719 (+ 641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:49.270885 (+ 166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:49.270935 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:49.278575 (+ 7640us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:46881: BlockingRecv error: recv got EOF from 127.0.98.1:46881 (error 108)
Metrics: {"server-negotiator.queue_time_us":89,"thread_start_us":65,"threads_started":1}
Sep 24 21:08:49 dist-test-slave-c7tk krb5kdc[8508](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 21:08:49 dist-test-slave-c7tk krb5kdc[8508](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 21:08:49.374653 9567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:49.370291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34181 (local address 127.0.98.2:50285)
0924 21:08:49.370429 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0924 21:08:49.370452 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0924 21:08:49.370503 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 21:08:49.370624 (+ 121us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 21:08:49.370624 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 21:08:49.370795 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0924 21:08:49.370840 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:49.370842 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:49.371485 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:49.371485 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:49.372001 (+ 516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:49.372002 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:49.372334 (+ 332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:49.372334 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:49.372375 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:49.372829 (+ 454us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 21:08:49.372833 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 21:08:49.374580 (+ 1747us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34181: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":86,"threads_started":1}
W20260924 21:08:49.374717 9566 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 21:08:49.370383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:50285 (local address 127.0.98.62:34181)
0924 21:08:49.370429 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0924 21:08:49.370431 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 21:08:49.370492 (+ 61us) server_negotiation.cc:407] Connection header received
0924 21:08:49.370547 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:08:49.370547 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:08:49.370556 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:08:49.370609 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:08:49.370907 (+ 298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:49.371390 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:49.372079 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:49.372268 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:49.372314 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:49.374674 (+ 2360us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:50285: BlockingRecv error: recv got EOF from 127.0.98.2:50285 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
I20260924 21:08:49.417338 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAzNNjS/build/release/bin/kudu with pid 9091
I20260924 21:08:49.422641 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAzNNjS/build/release/bin/kudu with pid 9243
I20260924 21:08:49.426582 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAzNNjS/build/release/bin/kudu with pid 9375
I20260924 21:08:49.430181 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAzNNjS/build/release/bin/kudu with pid 9021
2026-09-24T21:08:49Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (17137 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-24T21:08:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T21:08:49Z Disabled control of system clock
I20260924 21:08:49.455344 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAzNNjS/build/release/bin/kudu
/tmp/dist-test-taskAzNNjS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790284076800039-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790284076800039-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790284076800039-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790284076800039-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45003
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:35489
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45003
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260924 21:08:49.526197 9571 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 21:08:49.526371 9571 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 21:08:49.526389 9571 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 21:08:49.527462 9571 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 21:08:49.527498 9571 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 21:08:49.527510 9571 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 21:08:49.527522 9571 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 21:08:49.527566 9571 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 21:08:49.528833 9571 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35489
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790284076800039-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790284076800039-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45003
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45003
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790284076800039-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9571
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790284076800039-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 19:49:06 UTC on ccycloud.quasar-vhqjoc.root.comops.site
build id 4670864
I20260924 21:08:49.529038 9571 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 21:08:49.529297 9571 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 21:08:49.534220 9577 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 21:08:49.534291 9571 server_base.cc:1034] running on GCE node
W20260924 21:08:49.534205 9579 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 21:08:49.534197 9576 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 21:08:49.534646 9571 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 21:08:49.534919 9571 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 21:08:49.536096 9571 hybrid_clock.cc:648] HybridClock initialized: now 1790284129536075 us; error 32 us; skew 500 ppm
I20260924 21:08:49.537174 9571 webserver.cc:460] Webserver started at http://127.0.98.62:42111/ using document root <none> and password file <none>
I20260924 21:08:49.537416 9571 fs_manager.cc:359] Metadata directory not provided
I20260924 21:08:49.537497 9571 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 21:08:49.537628 9571 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 21:08:49.538635 9571 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790284076800039-392-0/minicluster-data/master-0/data/instance:
uuid: "4cbbca188408474cbc14135c11eb8774"
format_stamp: "Formatted at 2026-09-24 21:08:49 on dist-test-slave-c7tk"
server_key: "c5835057b340eb1825cd5662c0820b21"
server_key_iv: "9ed93aab5f578339c3139c3c0cb6f4f9"
server_key_version: "encryptionkey@0"
I20260924 21:08:49.538986 9571 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790284076800039-392-0/minicluster-data/master-0/wal/instance:
uuid: "4cbbca188408474cbc14135c11eb8774"
format_stamp: "Formatted at 2026-09-24 21:08:49 on dist-test-slave-c7tk"
server_key: "c5835057b340eb1825cd5662c0820b21"
server_key_iv: "9ed93aab5f578339c3139c3c0cb6f4f9"
server_key_version: "encryptionkey@0"
I20260924 21:08:49.540069 9571 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 21:08:49.540704 9585 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 21:08:49.541057 9571 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 21:08:49.541177 9571 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790284076800039-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790284076800039-392-0/minicluster-data/master-0/wal
uuid: "4cbbca188408474cbc14135c11eb8774"
format_stamp: "Formatted at 2026-09-24 21:08:49 on dist-test-slave-c7tk"
server_key: "c5835057b340eb1825cd5662c0820b21"
server_key_iv: "9ed93aab5f578339c3139c3c0cb6f4f9"
server_key_version: "encryptionkey@0"
I20260924 21:08:49.541266 9571 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790284076800039-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790284076800039-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790284076800039-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 21:08:49.552986 9571 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 21:08:49.553203 9571 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 21:08:49.553349 9571 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 21:08:49.556607 9571 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45003
I20260924 21:08:49.556636 9637 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45003 every 8 connection(s)
I20260924 21:08:49.556912 9571 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790284076800039-392-0/minicluster-data/master-0/data/info.pb
I20260924 21:08:49.557441 9638 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 21:08:49.558682 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAzNNjS/build/release/bin/kudu as pid 9571
I20260924 21:08:49.558804 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790284076800039-392-0/minicluster-data/master-0/wal/instance
I20260924 21:08:49.558990 392 external_mini_cluster.cc:1428] Setting key efa97a7d996ac1320fe77c48eaa8210b
I20260924 21:08:49.560005 9638 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4cbbca188408474cbc14135c11eb8774: Bootstrap starting.
I20260924 21:08:49.560592 9638 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4cbbca188408474cbc14135c11eb8774: Neither blocks nor log segments found. Creating new log.
I20260924 21:08:49.560864 9638 log.cc:824] T 00000000000000000000000000000000 P 4cbbca188408474cbc14135c11eb8774: Log is configured to *not* fsync() on all Append() calls
I20260924 21:08:49.561565 9638 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4cbbca188408474cbc14135c11eb8774: No bootstrap required, opened a new log
I20260924 21:08:49.563458 9638 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4cbbca188408474cbc14135c11eb8774 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4cbbca188408474cbc14135c11eb8774" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45003 } }
I20260924 21:08:49.563561 9638 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4cbbca188408474cbc14135c11eb8774 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 21:08:49.563580 9638 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4cbbca188408474cbc14135c11eb8774 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4cbbca188408474cbc14135c11eb8774, State: Initialized, Role: FOLLOWER
I20260924 21:08:49.563735 9638 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4cbbca188408474cbc14135c11eb8774 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4cbbca188408474cbc14135c11eb8774" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45003 } }
I20260924 21:08:49.563831 9638 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4cbbca188408474cbc14135c11eb8774 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 21:08:49.563860 9638 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4cbbca188408474cbc14135c11eb8774 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 21:08:49.563903 9638 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4cbbca188408474cbc14135c11eb8774 [term 0 FOLLOWER]: Advancing to term 1
I20260924 21:08:49.564270 9641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:49.559457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43670 (local address 127.0.98.62:45003)
0924 21:08:49.559653 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0924 21:08:49.559661 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 21:08:49.559688 (+ 27us) server_negotiation.cc:407] Connection header received
0924 21:08:49.560110 (+ 422us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:08:49.560117 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:08:49.560178 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:08:49.560326 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:08:49.560453 (+ 127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:49.561702 (+ 1249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:49.562492 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:49.563013 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:49.563139 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:49.563667 (+ 528us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 21:08:49.563671 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 21:08:49.563671 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 21:08:49.563846 (+ 175us) server_negotiation.cc:922] calling sasl_server_start()
0924 21:08:49.563885 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 21:08:49.563897 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 21:08:49.563957 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0924 21:08:49.564020 (+ 63us) server_negotiation.cc:299] Negotiation successful
0924 21:08:49.564084 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119,"thread_start_us":80,"threads_started":1}
I20260924 21:08:49.564460 9638 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4cbbca188408474cbc14135c11eb8774 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4cbbca188408474cbc14135c11eb8774" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45003 } }
I20260924 21:08:49.564560 9638 leader_election.cc:302] T 00000000000000000000000000000000 P 4cbbca188408474cbc14135c11eb8774 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4cbbca188408474cbc14135c11eb8774; no voters:
I20260924 21:08:49.564688 9638 leader_election.cc:288] T 00000000000000000000000000000000 P 4cbbca188408474cbc14135c11eb8774 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 21:08:49.564742 9643 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4cbbca188408474cbc14135c11eb8774 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 21:08:49.564828 9643 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4cbbca188408474cbc14135c11eb8774 [term 1 LEADER]: Becoming Leader. State: Replica: 4cbbca188408474cbc14135c11eb8774, State: Running, Role: LEADER
I20260924 21:08:49.564919 9638 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4cbbca188408474cbc14135c11eb8774 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 21:08:49.564939 9643 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4cbbca188408474cbc14135c11eb8774 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4cbbca188408474cbc14135c11eb8774" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45003 } }
I20260924 21:08:49.565181 9643 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4cbbca188408474cbc14135c11eb8774 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4cbbca188408474cbc14135c11eb8774" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4cbbca188408474cbc14135c11eb8774" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45003 } } }
I20260924 21:08:49.565254 9643 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4cbbca188408474cbc14135c11eb8774 [sys.catalog]: This master's current role is: LEADER
I20260924 21:08:49.565394 9643 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4cbbca188408474cbc14135c11eb8774 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4cbbca188408474cbc14135c11eb8774. Latest consensus state: current_term: 1 leader_uuid: "4cbbca188408474cbc14135c11eb8774" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4cbbca188408474cbc14135c11eb8774" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45003 } } }
I20260924 21:08:49.565469 9650 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 21:08:49.565518 9643 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4cbbca188408474cbc14135c11eb8774 [sys.catalog]: This master's current role is: LEADER
I20260924 21:08:49.566012 9650 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 21:08:49.567444 9650 catalog_manager.cc:1347] Generated new cluster ID: 22365a2cc4b346b1acd15b068c319856
I20260924 21:08:49.567508 9650 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 21:08:49.575050 9650 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 21:08:49.575515 9650 catalog_manager.cc:1504] Loading token signing keys...
I20260924 21:08:49.582466 9650 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4cbbca188408474cbc14135c11eb8774: Generated new TSK 0
I20260924 21:08:49.582612 9650 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 21:08:49.585451 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAzNNjS/build/release/bin/kudu
/tmp/dist-test-taskAzNNjS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790284076800039-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790284076800039-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790284076800039-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790284076800039-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45003
--builtin_ntp_servers=127.0.98.20:35489
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260924 21:08:49.658085 9662 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 21:08:49.658247 9662 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 21:08:49.658265 9662 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 21:08:49.659271 9662 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 21:08:49.659305 9662 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 21:08:49.659327 9662 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 21:08:49.660456 9662 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35489
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790284076800039-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790284076800039-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790284076800039-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45003
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9662
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790284076800039-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 19:49:06 UTC on ccycloud.quasar-vhqjoc.root.comops.site
build id 4670864
I20260924 21:08:49.660647 9662 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 21:08:49.660912 9662 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 21:08:49.666031 9668 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 21:08:49.666077 9667 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 21:08:49.666072 9670 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 21:08:49.666239 9662 server_base.cc:1034] running on GCE node
I20260924 21:08:49.666563 9662 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 21:08:49.666801 9662 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 21:08:49.667934 9662 hybrid_clock.cc:648] HybridClock initialized: now 1790284129667918 us; error 31 us; skew 500 ppm
I20260924 21:08:49.669049 9662 webserver.cc:460] Webserver started at http://127.0.98.1:40483/ using document root <none> and password file <none>
I20260924 21:08:49.669253 9662 fs_manager.cc:359] Metadata directory not provided
I20260924 21:08:49.669315 9662 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 21:08:49.669435 9662 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 21:08:49.670617 9662 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790284076800039-392-0/minicluster-data/ts-0/data/instance:
uuid: "bf7e8cd7350b466ca217000cfa8317e1"
format_stamp: "Formatted at 2026-09-24 21:08:49 on dist-test-slave-c7tk"
server_key: "9556d52028b732286771b6139ec470bb"
server_key_iv: "e2c682a5cc5ee0a2e565c476327c556a"
server_key_version: "encryptionkey@0"
I20260924 21:08:49.670975 9662 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790284076800039-392-0/minicluster-data/ts-0/wal/instance:
uuid: "bf7e8cd7350b466ca217000cfa8317e1"
format_stamp: "Formatted at 2026-09-24 21:08:49 on dist-test-slave-c7tk"
server_key: "9556d52028b732286771b6139ec470bb"
server_key_iv: "e2c682a5cc5ee0a2e565c476327c556a"
server_key_version: "encryptionkey@0"
I20260924 21:08:49.672256 9662 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260924 21:08:49.672960 9676 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 21:08:49.673097 9662 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 21:08:49.673169 9662 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790284076800039-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790284076800039-392-0/minicluster-data/ts-0/wal
uuid: "bf7e8cd7350b466ca217000cfa8317e1"
format_stamp: "Formatted at 2026-09-24 21:08:49 on dist-test-slave-c7tk"
server_key: "9556d52028b732286771b6139ec470bb"
server_key_iv: "e2c682a5cc5ee0a2e565c476327c556a"
server_key_version: "encryptionkey@0"
I20260924 21:08:49.673228 9662 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790284076800039-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790284076800039-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790284076800039-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 21:08:49.695362 9662 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 21:08:49.695598 9662 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 21:08:49.695739 9662 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 21:08:49.695927 9662 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 21:08:49.696233 9662 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 21:08:49.696266 9662 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 21:08:49.696314 9662 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 21:08:49.696338 9662 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 21:08:49.701622 9662 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:38141
I20260924 21:08:49.701658 9789 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:38141 every 8 connection(s)
I20260924 21:08:49.701953 9662 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790284076800039-392-0/minicluster-data/ts-0/data/info.pb
I20260924 21:08:49.706426 9641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:49.702609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:45809 (local address 127.0.98.62:45003)
0924 21:08:49.702663 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0924 21:08:49.702664 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 21:08:49.702897 (+ 233us) server_negotiation.cc:407] Connection header received
0924 21:08:49.703339 (+ 442us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:08:49.703340 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:08:49.703351 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:08:49.703413 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:08:49.703940 (+ 527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:49.704356 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:49.705214 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:49.705384 (+ 170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:49.705452 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:49.706139 (+ 687us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 21:08:49.706140 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 21:08:49.706140 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 21:08:49.706145 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 21:08:49.706156 (+ 11us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 21:08:49.706160 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 21:08:49.706205 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0924 21:08:49.706320 (+ 115us) server_negotiation.cc:299] Negotiation successful
0924 21:08:49.706342 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260924 21:08:49.706604 9792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:49.702593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45003 (local address 127.0.98.1:45809)
0924 21:08:49.702795 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0924 21:08:49.702821 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0924 21:08:49.703245 (+ 424us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 21:08:49.703466 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 21:08:49.703469 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 21:08:49.703481 (+ 12us) client_negotiation.cc:189] Negotiated authn=SASL
0924 21:08:49.703858 (+ 377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:49.703862 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:49.704438 (+ 576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:49.704439 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:49.705120 (+ 681us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:49.705122 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:49.705468 (+ 346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:49.705468 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:49.705579 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:49.706018 (+ 439us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 21:08:49.706018 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0924 21:08:49.706041 (+ 23us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 21:08:49.706059 (+ 18us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 21:08:49.706204 (+ 145us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 21:08:49.706207 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 21:08:49.706210 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 21:08:49.706309 (+ 99us) client_negotiation.cc:240] Negotiation successful
0924 21:08:49.706419 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":101,"threads_started":1}
I20260924 21:08:49.707050 9790 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45003
I20260924 21:08:49.707129 9790 heartbeater.cc:464] Registering TS with master...
I20260924 21:08:49.707275 9790 heartbeater.cc:511] Master 127.0.98.62:45003 requested a full tablet report, sending...
I20260924 21:08:49.707679 9602 ts_manager.cc:194] Registered new tserver with Master: bf7e8cd7350b466ca217000cfa8317e1 (127.0.98.1:38141)
I20260924 21:08:49.708168 9602 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:45809
I20260924 21:08:49.710070 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAzNNjS/build/release/bin/kudu as pid 9662
I20260924 21:08:49.710131 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790284076800039-392-0/minicluster-data/ts-0/wal/instance
I20260924 21:08:49.710286 392 external_mini_cluster.cc:1428] Setting key bf7cff0a029d18024d5b9c39b4ee5a91
I20260924 21:08:49.710923 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAzNNjS/build/release/bin/kudu
/tmp/dist-test-taskAzNNjS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790284076800039-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790284076800039-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790284076800039-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790284076800039-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45003
--builtin_ntp_servers=127.0.98.20:35489
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260924 21:08:49.781105 9793 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 21:08:49.781309 9793 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 21:08:49.781329 9793 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 21:08:49.782474 9793 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 21:08:49.782509 9793 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 21:08:49.782529 9793 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 21:08:49.783602 9793 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35489
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790284076800039-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790284076800039-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790284076800039-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45003
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9793
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790284076800039-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 19:49:06 UTC on ccycloud.quasar-vhqjoc.root.comops.site
build id 4670864
I20260924 21:08:49.783794 9793 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 21:08:49.784022 9793 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 21:08:49.788270 9799 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 21:08:49.788365 9793 server_base.cc:1034] running on GCE node
W20260924 21:08:49.788231 9801 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 21:08:49.788296 9798 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 21:08:49.788638 9793 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 21:08:49.788890 9793 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 21:08:49.790052 9793 hybrid_clock.cc:648] HybridClock initialized: now 1790284129790035 us; error 34 us; skew 500 ppm
I20260924 21:08:49.791148 9793 webserver.cc:460] Webserver started at http://127.0.98.2:38093/ using document root <none> and password file <none>
I20260924 21:08:49.791363 9793 fs_manager.cc:359] Metadata directory not provided
I20260924 21:08:49.791428 9793 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 21:08:49.791566 9793 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 21:08:49.792526 9793 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790284076800039-392-0/minicluster-data/ts-1/data/instance:
uuid: "7bf4511c3bd04a149c09a232e509e0a2"
format_stamp: "Formatted at 2026-09-24 21:08:49 on dist-test-slave-c7tk"
server_key: "d23d319cc1d8a7db76b5da7c8e7046d7"
server_key_iv: "dceecb547a5372eded8488aa82d0292a"
server_key_version: "encryptionkey@0"
I20260924 21:08:49.792862 9793 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790284076800039-392-0/minicluster-data/ts-1/wal/instance:
uuid: "7bf4511c3bd04a149c09a232e509e0a2"
format_stamp: "Formatted at 2026-09-24 21:08:49 on dist-test-slave-c7tk"
server_key: "d23d319cc1d8a7db76b5da7c8e7046d7"
server_key_iv: "dceecb547a5372eded8488aa82d0292a"
server_key_version: "encryptionkey@0"
I20260924 21:08:49.794047 9793 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 21:08:49.794695 9807 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 21:08:49.794840 9793 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 21:08:49.794920 9793 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790284076800039-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790284076800039-392-0/minicluster-data/ts-1/wal
uuid: "7bf4511c3bd04a149c09a232e509e0a2"
format_stamp: "Formatted at 2026-09-24 21:08:49 on dist-test-slave-c7tk"
server_key: "d23d319cc1d8a7db76b5da7c8e7046d7"
server_key_iv: "dceecb547a5372eded8488aa82d0292a"
server_key_version: "encryptionkey@0"
I20260924 21:08:49.794987 9793 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790284076800039-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790284076800039-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790284076800039-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 21:08:49.800153 9793 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 21:08:49.800386 9793 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 21:08:49.800514 9793 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 21:08:49.800717 9793 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 21:08:49.800982 9793 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 21:08:49.801030 9793 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 21:08:49.801065 9793 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 21:08:49.801095 9793 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 21:08:49.806464 9793 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:40235
I20260924 21:08:49.806499 9920 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:40235 every 8 connection(s)
I20260924 21:08:49.806758 9793 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790284076800039-392-0/minicluster-data/ts-1/data/info.pb
I20260924 21:08:49.811292 9641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:49.807302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:54333 (local address 127.0.98.62:45003)
0924 21:08:49.807338 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0924 21:08:49.807340 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 21:08:49.807576 (+ 236us) server_negotiation.cc:407] Connection header received
0924 21:08:49.808041 (+ 465us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:08:49.808042 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:08:49.808055 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:08:49.808110 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:08:49.808720 (+ 610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:49.809165 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:49.810085 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:49.810260 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:49.810341 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:49.810995 (+ 654us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 21:08:49.810996 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 21:08:49.810997 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 21:08:49.811002 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 21:08:49.811014 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 21:08:49.811018 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 21:08:49.811073 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0924 21:08:49.811188 (+ 115us) server_negotiation.cc:299] Negotiation successful
0924 21:08:49.811213 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260924 21:08:49.811488 9923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:49.807303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45003 (local address 127.0.98.2:54333)
0924 21:08:49.807463 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0924 21:08:49.807487 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0924 21:08:49.807937 (+ 450us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 21:08:49.808162 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 21:08:49.808166 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 21:08:49.808178 (+ 12us) client_negotiation.cc:189] Negotiated authn=SASL
0924 21:08:49.808589 (+ 411us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:49.808592 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:49.809270 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:49.809271 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:49.809991 (+ 720us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:49.809993 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:49.810358 (+ 365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:49.810359 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:49.810467 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:49.810841 (+ 374us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 21:08:49.810841 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0924 21:08:49.810880 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 21:08:49.810898 (+ 18us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 21:08:49.811078 (+ 180us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 21:08:49.811083 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 21:08:49.811086 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 21:08:49.811186 (+ 100us) client_negotiation.cc:240] Negotiation successful
0924 21:08:49.811277 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":66,"threads_started":1}
I20260924 21:08:49.811967 9921 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45003
I20260924 21:08:49.812052 9921 heartbeater.cc:464] Registering TS with master...
I20260924 21:08:49.812186 9921 heartbeater.cc:511] Master 127.0.98.62:45003 requested a full tablet report, sending...
I20260924 21:08:49.812518 9602 ts_manager.cc:194] Registered new tserver with Master: 7bf4511c3bd04a149c09a232e509e0a2 (127.0.98.2:40235)
I20260924 21:08:49.812824 9602 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:54333
I20260924 21:08:49.814749 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAzNNjS/build/release/bin/kudu as pid 9793
I20260924 21:08:49.814816 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790284076800039-392-0/minicluster-data/ts-1/wal/instance
I20260924 21:08:49.815011 392 external_mini_cluster.cc:1428] Setting key f8171bb6ebf28df15c9ff056a45a6cfd
I20260924 21:08:49.815860 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAzNNjS/build/release/bin/kudu
/tmp/dist-test-taskAzNNjS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790284076800039-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790284076800039-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790284076800039-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790284076800039-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45003
--builtin_ntp_servers=127.0.98.20:35489
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260924 21:08:49.885403 9924 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 21:08:49.885556 9924 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 21:08:49.885573 9924 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 21:08:49.886565 9924 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 21:08:49.886601 9924 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 21:08:49.886621 9924 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 21:08:49.887735 9924 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35489
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790284076800039-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790284076800039-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790284076800039-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45003
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9924
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790284076800039-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 19:49:06 UTC on ccycloud.quasar-vhqjoc.root.comops.site
build id 4670864
I20260924 21:08:49.887930 9924 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 21:08:49.888175 9924 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 21:08:49.892660 9932 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 21:08:49.892728 9929 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 21:08:49.892669 9930 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 21:08:49.892874 9924 server_base.cc:1034] running on GCE node
I20260924 21:08:49.893043 9924 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 21:08:49.893200 9924 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 21:08:49.894328 9924 hybrid_clock.cc:648] HybridClock initialized: now 1790284129894313 us; error 34 us; skew 500 ppm
I20260924 21:08:49.895418 9924 webserver.cc:460] Webserver started at http://127.0.98.3:33227/ using document root <none> and password file <none>
I20260924 21:08:49.895659 9924 fs_manager.cc:359] Metadata directory not provided
I20260924 21:08:49.895699 9924 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 21:08:49.895820 9924 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 21:08:49.896706 9924 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790284076800039-392-0/minicluster-data/ts-2/data/instance:
uuid: "6e3d9d5f60874f6aa79eec51c2157506"
format_stamp: "Formatted at 2026-09-24 21:08:49 on dist-test-slave-c7tk"
server_key: "3d5df444e29cdd47c16500cbe54a61c4"
server_key_iv: "2c1bbaa234f5a42db66c952d4f85f0c1"
server_key_version: "encryptionkey@0"
I20260924 21:08:49.897017 9924 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790284076800039-392-0/minicluster-data/ts-2/wal/instance:
uuid: "6e3d9d5f60874f6aa79eec51c2157506"
format_stamp: "Formatted at 2026-09-24 21:08:49 on dist-test-slave-c7tk"
server_key: "3d5df444e29cdd47c16500cbe54a61c4"
server_key_iv: "2c1bbaa234f5a42db66c952d4f85f0c1"
server_key_version: "encryptionkey@0"
I20260924 21:08:49.898242 9924 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 21:08:49.898880 9938 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 21:08:49.899031 9924 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 21:08:49.899114 9924 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790284076800039-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790284076800039-392-0/minicluster-data/ts-2/wal
uuid: "6e3d9d5f60874f6aa79eec51c2157506"
format_stamp: "Formatted at 2026-09-24 21:08:49 on dist-test-slave-c7tk"
server_key: "3d5df444e29cdd47c16500cbe54a61c4"
server_key_iv: "2c1bbaa234f5a42db66c952d4f85f0c1"
server_key_version: "encryptionkey@0"
I20260924 21:08:49.899188 9924 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790284076800039-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790284076800039-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790284076800039-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 21:08:49.917268 9924 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 21:08:49.917495 9924 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 21:08:49.917616 9924 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 21:08:49.917834 9924 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 21:08:49.918087 9924 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 21:08:49.918133 9924 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 21:08:49.918160 9924 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 21:08:49.918183 9924 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 21:08:49.923319 9924 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:33787
I20260924 21:08:49.923362 10051 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:33787 every 8 connection(s)
I20260924 21:08:49.923650 9924 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790284076800039-392-0/minicluster-data/ts-2/data/info.pb
I20260924 21:08:49.928025 9641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:49.924302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:34957 (local address 127.0.98.62:45003)
0924 21:08:49.924354 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0924 21:08:49.924356 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 21:08:49.924542 (+ 186us) server_negotiation.cc:407] Connection header received
0924 21:08:49.925029 (+ 487us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:08:49.925030 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:08:49.925038 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:08:49.925098 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:08:49.925636 (+ 538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:49.926103 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:49.926923 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:49.927086 (+ 163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:49.927135 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:49.927854 (+ 719us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 21:08:49.927855 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 21:08:49.927856 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 21:08:49.927859 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0924 21:08:49.927868 (+ 9us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 21:08:49.927871 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 21:08:49.927905 (+ 34us) server_negotiation.cc:1035] Waiting for connection context
0924 21:08:49.927960 (+ 55us) server_negotiation.cc:299] Negotiation successful
0924 21:08:49.927975 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 21:08:49.928216 10054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:49.924277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45003 (local address 127.0.98.3:34957)
0924 21:08:49.924444 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0924 21:08:49.924485 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 21:08:49.924935 (+ 450us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 21:08:49.925144 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 21:08:49.925148 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 21:08:49.925155 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0924 21:08:49.925576 (+ 421us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:49.925579 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:49.926164 (+ 585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:49.926165 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:49.926847 (+ 682us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:49.926848 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:49.927150 (+ 302us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:49.927151 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:49.927259 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:49.927733 (+ 474us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 21:08:49.927733 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0924 21:08:49.927768 (+ 35us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 21:08:49.927787 (+ 19us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 21:08:49.927905 (+ 118us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 21:08:49.927907 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 21:08:49.927909 (+ 2us) client_negotiation.cc:769] Sending connection context
0924 21:08:49.927949 (+ 40us) client_negotiation.cc:240] Negotiation successful
0924 21:08:49.928011 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":83,"threads_started":1}
I20260924 21:08:49.928838 10052 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45003
I20260924 21:08:49.928918 10052 heartbeater.cc:464] Registering TS with master...
I20260924 21:08:49.929034 10052 heartbeater.cc:511] Master 127.0.98.62:45003 requested a full tablet report, sending...
I20260924 21:08:49.929355 9602 ts_manager.cc:194] Registered new tserver with Master: 6e3d9d5f60874f6aa79eec51c2157506 (127.0.98.3:33787)
I20260924 21:08:49.929656 9602 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:34957
I20260924 21:08:49.929730 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAzNNjS/build/release/bin/kudu as pid 9924
I20260924 21:08:49.929845 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790284076800039-392-0/minicluster-data/ts-2/wal/instance
I20260924 21:08:49.930047 392 external_mini_cluster.cc:1428] Setting key 1777de6ec8b6f76deb4f2ae1cf604bee
I20260924 21:08:49.930866 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260924 21:08:49.932480 9641 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 21:08:49.932076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43672 (local address 127.0.98.62:45003)
0924 21:08:49.932130 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0924 21:08:49.932132 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 21:08:49.932228 (+ 96us) server_negotiation.cc:407] Connection header received
0924 21:08:49.932280 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:08:49.932280 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:08:49.932300 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:08:49.932340 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:08:49.932444 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43672: BlockingRecv error: recv got EOF from 127.0.0.1:43672 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
I20260924 21:08:49.932557 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAzNNjS/build/release/bin/kudu with pid 9662
I20260924 21:08:49.937216 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAzNNjS/build/release/bin/kudu with pid 9793
I20260924 21:08:49.940761 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAzNNjS/build/release/bin/kudu with pid 9924
I20260924 21:08:49.944584 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAzNNjS/build/release/bin/kudu with pid 9571
2026-09-24T21:08:49Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (514 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-24T21:08:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T21:08:49Z Disabled control of system clock
I20260924 21:08:49.969229 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAzNNjS/build/release/bin/kudu
/tmp/dist-test-taskAzNNjS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790284076800039-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790284076800039-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790284076800039-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790284076800039-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33683
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:37441
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33683
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260924 21:08:50.038583 10063 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 21:08:50.038738 10063 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 21:08:50.038754 10063 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 21:08:50.040025 10063 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 21:08:50.040074 10063 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 21:08:50.040107 10063 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 21:08:50.040128 10063 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 21:08:50.040158 10063 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 21:08:50.042405 10063 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37441
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790284076800039-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790284076800039-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33683
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33683
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790284076800039-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10063
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790284076800039-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 19:49:06 UTC on ccycloud.quasar-vhqjoc.root.comops.site
build id 4670864
I20260924 21:08:50.042629 10063 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 21:08:50.042927 10063 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 21:08:50.047921 10069 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 21:08:50.047966 10063 server_base.cc:1034] running on GCE node
W20260924 21:08:50.047930 10068 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 21:08:50.047940 10071 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 21:08:50.048265 10063 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 21:08:50.048554 10063 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 21:08:50.049690 10063 hybrid_clock.cc:648] HybridClock initialized: now 1790284130049678 us; error 25 us; skew 500 ppm
I20260924 21:08:50.050722 10063 webserver.cc:460] Webserver started at http://127.0.98.62:33733/ using document root <none> and password file <none>
I20260924 21:08:50.050941 10063 fs_manager.cc:359] Metadata directory not provided
I20260924 21:08:50.050979 10063 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 21:08:50.051096 10063 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 21:08:50.052048 10063 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790284076800039-392-0/minicluster-data/master-0/data/instance:
uuid: "1dba688645dd4944945edc86a05d6607"
format_stamp: "Formatted at 2026-09-24 21:08:50 on dist-test-slave-c7tk"
server_key: "3b094d19e4b4d4f9d830f781c9f338cf"
server_key_iv: "25312528efe6841a97284c08ffa5c7c7"
server_key_version: "encryptionkey@0"
I20260924 21:08:50.052421 10063 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790284076800039-392-0/minicluster-data/master-0/wal/instance:
uuid: "1dba688645dd4944945edc86a05d6607"
format_stamp: "Formatted at 2026-09-24 21:08:50 on dist-test-slave-c7tk"
server_key: "3b094d19e4b4d4f9d830f781c9f338cf"
server_key_iv: "25312528efe6841a97284c08ffa5c7c7"
server_key_version: "encryptionkey@0"
I20260924 21:08:50.053478 10063 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 21:08:50.054155 10077 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 21:08:50.054302 10063 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 21:08:50.054375 10063 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790284076800039-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790284076800039-392-0/minicluster-data/master-0/wal
uuid: "1dba688645dd4944945edc86a05d6607"
format_stamp: "Formatted at 2026-09-24 21:08:50 on dist-test-slave-c7tk"
server_key: "3b094d19e4b4d4f9d830f781c9f338cf"
server_key_iv: "25312528efe6841a97284c08ffa5c7c7"
server_key_version: "encryptionkey@0"
I20260924 21:08:50.054445 10063 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790284076800039-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790284076800039-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790284076800039-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 21:08:50.057163 10063 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 21:08:50.057385 10063 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 21:08:50.057518 10063 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 21:08:50.060878 10063 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33683
I20260924 21:08:50.060923 10129 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33683 every 8 connection(s)
I20260924 21:08:50.061225 10063 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790284076800039-392-0/minicluster-data/master-0/data/info.pb
I20260924 21:08:50.061900 10130 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 21:08:50.062672 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAzNNjS/build/release/bin/kudu as pid 10063
I20260924 21:08:50.062754 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790284076800039-392-0/minicluster-data/master-0/wal/instance
I20260924 21:08:50.062942 392 external_mini_cluster.cc:1428] Setting key 11236733ce9efed3f21addabe3d912e5
I20260924 21:08:50.064713 10130 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1dba688645dd4944945edc86a05d6607: Bootstrap starting.
I20260924 21:08:50.065070 10133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:50.063496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60074 (local address 127.0.98.62:33683)
0924 21:08:50.063652 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0924 21:08:50.063657 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 21:08:50.063681 (+ 24us) server_negotiation.cc:407] Connection header received
0924 21:08:50.064005 (+ 324us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:08:50.064010 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:08:50.064041 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:08:50.064137 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:08:50.064702 (+ 565us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 21:08:50.064705 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 21:08:50.064705 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 21:08:50.064709 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 21:08:50.064742 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 21:08:50.064750 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 21:08:50.064833 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0924 21:08:50.064901 (+ 68us) server_negotiation.cc:299] Negotiation successful
0924 21:08:50.064949 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101,"thread_start_us":64,"threads_started":1}
I20260924 21:08:50.065342 10130 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1dba688645dd4944945edc86a05d6607: Neither blocks nor log segments found. Creating new log.
I20260924 21:08:50.065582 10130 log.cc:824] T 00000000000000000000000000000000 P 1dba688645dd4944945edc86a05d6607: Log is configured to *not* fsync() on all Append() calls
I20260924 21:08:50.066179 10130 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1dba688645dd4944945edc86a05d6607: No bootstrap required, opened a new log
I20260924 21:08:50.067200 10130 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1dba688645dd4944945edc86a05d6607 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1dba688645dd4944945edc86a05d6607" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33683 } }
I20260924 21:08:50.067304 10130 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1dba688645dd4944945edc86a05d6607 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 21:08:50.067323 10130 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1dba688645dd4944945edc86a05d6607 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1dba688645dd4944945edc86a05d6607, State: Initialized, Role: FOLLOWER
I20260924 21:08:50.067418 10130 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1dba688645dd4944945edc86a05d6607 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1dba688645dd4944945edc86a05d6607" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33683 } }
I20260924 21:08:50.067492 10130 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1dba688645dd4944945edc86a05d6607 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 21:08:50.067512 10130 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1dba688645dd4944945edc86a05d6607 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 21:08:50.067534 10130 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1dba688645dd4944945edc86a05d6607 [term 0 FOLLOWER]: Advancing to term 1
I20260924 21:08:50.067909 10130 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1dba688645dd4944945edc86a05d6607 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1dba688645dd4944945edc86a05d6607" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33683 } }
I20260924 21:08:50.067975 10130 leader_election.cc:302] T 00000000000000000000000000000000 P 1dba688645dd4944945edc86a05d6607 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1dba688645dd4944945edc86a05d6607; no voters:
I20260924 21:08:50.068066 10130 leader_election.cc:288] T 00000000000000000000000000000000 P 1dba688645dd4944945edc86a05d6607 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 21:08:50.068287 10130 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1dba688645dd4944945edc86a05d6607 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 21:08:50.068313 10135 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1dba688645dd4944945edc86a05d6607 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 21:08:50.068451 10135 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1dba688645dd4944945edc86a05d6607 [term 1 LEADER]: Becoming Leader. State: Replica: 1dba688645dd4944945edc86a05d6607, State: Running, Role: LEADER
I20260924 21:08:50.068537 10135 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1dba688645dd4944945edc86a05d6607 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1dba688645dd4944945edc86a05d6607" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33683 } }
I20260924 21:08:50.068961 10136 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1dba688645dd4944945edc86a05d6607 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1dba688645dd4944945edc86a05d6607" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1dba688645dd4944945edc86a05d6607" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33683 } } }
I20260924 21:08:50.069051 10136 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1dba688645dd4944945edc86a05d6607 [sys.catalog]: This master's current role is: LEADER
I20260924 21:08:50.069180 10137 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1dba688645dd4944945edc86a05d6607 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1dba688645dd4944945edc86a05d6607. Latest consensus state: current_term: 1 leader_uuid: "1dba688645dd4944945edc86a05d6607" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1dba688645dd4944945edc86a05d6607" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33683 } } }
I20260924 21:08:50.069478 10137 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1dba688645dd4944945edc86a05d6607 [sys.catalog]: This master's current role is: LEADER
I20260924 21:08:50.069607 10145 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 21:08:50.070021 10145 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 21:08:50.074154 10145 catalog_manager.cc:1347] Generated new cluster ID: 173219b0f20041cfbc8e0cd7ec05fc8e
I20260924 21:08:50.074218 10145 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 21:08:50.093937 10145 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 21:08:50.094066 10145 catalog_manager.cc:1504] Loading token signing keys...
I20260924 21:08:50.097358 10145 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1dba688645dd4944945edc86a05d6607: Generated new TSK 0
I20260924 21:08:50.097515 10145 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 21:08:50.100930 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAzNNjS/build/release/bin/kudu
/tmp/dist-test-taskAzNNjS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790284076800039-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790284076800039-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790284076800039-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790284076800039-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33683
--builtin_ntp_servers=127.0.98.20:37441
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260924 21:08:50.171604 10154 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 21:08:50.171818 10154 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 21:08:50.171855 10154 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 21:08:50.172899 10154 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 21:08:50.172959 10154 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 21:08:50.172996 10154 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 21:08:50.174152 10154 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37441
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790284076800039-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790284076800039-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790284076800039-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33683
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10154
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790284076800039-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 19:49:06 UTC on ccycloud.quasar-vhqjoc.root.comops.site
build id 4670864
I20260924 21:08:50.174369 10154 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 21:08:50.174604 10154 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 21:08:50.179721 10162 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 21:08:50.179705 10160 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 21:08:50.179848 10159 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 21:08:50.179862 10154 server_base.cc:1034] running on GCE node
I20260924 21:08:50.180111 10154 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 21:08:50.180352 10154 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 21:08:50.181499 10154 hybrid_clock.cc:648] HybridClock initialized: now 1790284130181489 us; error 24 us; skew 500 ppm
I20260924 21:08:50.182771 10154 webserver.cc:460] Webserver started at http://127.0.98.1:37165/ using document root <none> and password file <none>
I20260924 21:08:50.182984 10154 fs_manager.cc:359] Metadata directory not provided
I20260924 21:08:50.183023 10154 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 21:08:50.183141 10154 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 21:08:50.184051 10154 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790284076800039-392-0/minicluster-data/ts-0/data/instance:
uuid: "83d85f219d784130b4ef655b21bb237b"
format_stamp: "Formatted at 2026-09-24 21:08:50 on dist-test-slave-c7tk"
server_key: "d1d6bbbebe4e5ef8eab15c385b1fc1f4"
server_key_iv: "314a8adc4d48574611e8c6cb1d0bd87a"
server_key_version: "encryptionkey@0"
I20260924 21:08:50.184368 10154 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790284076800039-392-0/minicluster-data/ts-0/wal/instance:
uuid: "83d85f219d784130b4ef655b21bb237b"
format_stamp: "Formatted at 2026-09-24 21:08:50 on dist-test-slave-c7tk"
server_key: "d1d6bbbebe4e5ef8eab15c385b1fc1f4"
server_key_iv: "314a8adc4d48574611e8c6cb1d0bd87a"
server_key_version: "encryptionkey@0"
I20260924 21:08:50.185530 10154 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 21:08:50.186174 10168 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 21:08:50.186345 10154 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 21:08:50.186427 10154 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790284076800039-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790284076800039-392-0/minicluster-data/ts-0/wal
uuid: "83d85f219d784130b4ef655b21bb237b"
format_stamp: "Formatted at 2026-09-24 21:08:50 on dist-test-slave-c7tk"
server_key: "d1d6bbbebe4e5ef8eab15c385b1fc1f4"
server_key_iv: "314a8adc4d48574611e8c6cb1d0bd87a"
server_key_version: "encryptionkey@0"
I20260924 21:08:50.186501 10154 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790284076800039-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790284076800039-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790284076800039-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 21:08:50.189215 10154 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 21:08:50.189463 10154 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 21:08:50.189591 10154 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 21:08:50.189844 10154 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 21:08:50.190100 10154 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 21:08:50.190150 10154 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 21:08:50.190183 10154 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 21:08:50.190208 10154 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 21:08:50.195647 10154 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:33801
I20260924 21:08:50.195681 10281 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:33801 every 8 connection(s)
I20260924 21:08:50.195930 10154 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790284076800039-392-0/minicluster-data/ts-0/data/info.pb
I20260924 21:08:50.198151 10133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:50.196481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:34607 (local address 127.0.98.62:33683)
0924 21:08:50.196535 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0924 21:08:50.196537 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 21:08:50.196719 (+ 182us) server_negotiation.cc:407] Connection header received
0924 21:08:50.197028 (+ 309us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:08:50.197028 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:08:50.197032 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:08:50.197082 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:08:50.197996 (+ 914us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 21:08:50.197997 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 21:08:50.197997 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 21:08:50.198000 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0924 21:08:50.198011 (+ 11us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 21:08:50.198014 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 21:08:50.198058 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0924 21:08:50.198100 (+ 42us) server_negotiation.cc:299] Negotiation successful
0924 21:08:50.198109 (+ 9us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260924 21:08:50.198294 10284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:50.196470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33683 (local address 127.0.98.1:34607)
0924 21:08:50.196608 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0924 21:08:50.196638 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0924 21:08:50.196962 (+ 324us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 21:08:50.197122 (+ 160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 21:08:50.197124 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 21:08:50.197131 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0924 21:08:50.197890 (+ 759us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 21:08:50.197890 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0924 21:08:50.197911 (+ 21us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 21:08:50.197931 (+ 20us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 21:08:50.198058 (+ 127us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 21:08:50.198060 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 21:08:50.198062 (+ 2us) client_negotiation.cc:769] Sending connection context
0924 21:08:50.198099 (+ 37us) client_negotiation.cc:240] Negotiation successful
0924 21:08:50.198134 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":89,"thread_start_us":64,"threads_started":1}
I20260924 21:08:50.198716 10282 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33683
I20260924 21:08:50.198807 10282 heartbeater.cc:464] Registering TS with master...
I20260924 21:08:50.198923 10282 heartbeater.cc:511] Master 127.0.98.62:33683 requested a full tablet report, sending...
I20260924 21:08:50.199308 10094 ts_manager.cc:194] Registered new tserver with Master: 83d85f219d784130b4ef655b21bb237b (127.0.98.1:33801)
I20260924 21:08:50.204907 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAzNNjS/build/release/bin/kudu as pid 10154
I20260924 21:08:50.204990 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790284076800039-392-0/minicluster-data/ts-0/wal/instance
I20260924 21:08:50.205147 392 external_mini_cluster.cc:1428] Setting key fbfc9194946474d2c09b76127135ebde
I20260924 21:08:50.206033 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAzNNjS/build/release/bin/kudu
/tmp/dist-test-taskAzNNjS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790284076800039-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790284076800039-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790284076800039-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790284076800039-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33683
--builtin_ntp_servers=127.0.98.20:37441
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260924 21:08:50.277374 10285 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 21:08:50.277551 10285 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 21:08:50.277572 10285 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 21:08:50.278689 10285 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 21:08:50.278728 10285 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 21:08:50.278760 10285 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 21:08:50.279877 10285 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37441
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790284076800039-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790284076800039-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790284076800039-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33683
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10285
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790284076800039-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 19:49:06 UTC on ccycloud.quasar-vhqjoc.root.comops.site
build id 4670864
I20260924 21:08:50.280020 10285 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 21:08:50.280213 10285 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 21:08:50.284525 10291 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 21:08:50.284566 10293 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 21:08:50.284533 10290 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 21:08:50.284713 10285 server_base.cc:1034] running on GCE node
I20260924 21:08:50.284912 10285 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 21:08:50.285128 10285 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 21:08:50.286263 10285 hybrid_clock.cc:648] HybridClock initialized: now 1790284130286251 us; error 26 us; skew 500 ppm
I20260924 21:08:50.287304 10285 webserver.cc:460] Webserver started at http://127.0.98.2:46231/ using document root <none> and password file <none>
I20260924 21:08:50.287460 10285 fs_manager.cc:359] Metadata directory not provided
I20260924 21:08:50.287496 10285 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 21:08:50.287571 10285 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 21:08:50.288463 10285 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790284076800039-392-0/minicluster-data/ts-1/data/instance:
uuid: "d672fdfff72947dc8ccb2d2fbaf7ac1e"
format_stamp: "Formatted at 2026-09-24 21:08:50 on dist-test-slave-c7tk"
server_key: "7a7a072875cac56d961db650922ecb52"
server_key_iv: "dd5f3ef5df50640983eaa529797d0d52"
server_key_version: "encryptionkey@0"
I20260924 21:08:50.288710 10285 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790284076800039-392-0/minicluster-data/ts-1/wal/instance:
uuid: "d672fdfff72947dc8ccb2d2fbaf7ac1e"
format_stamp: "Formatted at 2026-09-24 21:08:50 on dist-test-slave-c7tk"
server_key: "7a7a072875cac56d961db650922ecb52"
server_key_iv: "dd5f3ef5df50640983eaa529797d0d52"
server_key_version: "encryptionkey@0"
I20260924 21:08:50.289892 10285 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 21:08:50.290498 10299 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 21:08:50.290684 10285 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 21:08:50.290753 10285 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790284076800039-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790284076800039-392-0/minicluster-data/ts-1/wal
uuid: "d672fdfff72947dc8ccb2d2fbaf7ac1e"
format_stamp: "Formatted at 2026-09-24 21:08:50 on dist-test-slave-c7tk"
server_key: "7a7a072875cac56d961db650922ecb52"
server_key_iv: "dd5f3ef5df50640983eaa529797d0d52"
server_key_version: "encryptionkey@0"
I20260924 21:08:50.290839 10285 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790284076800039-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790284076800039-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790284076800039-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 21:08:50.293853 10285 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 21:08:50.294085 10285 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 21:08:50.294193 10285 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 21:08:50.294373 10285 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 21:08:50.294616 10285 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 21:08:50.294643 10285 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 21:08:50.294689 10285 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 21:08:50.294714 10285 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 21:08:50.299515 10285 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:41141
I20260924 21:08:50.299575 10412 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:41141 every 8 connection(s)
I20260924 21:08:50.299811 10285 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790284076800039-392-0/minicluster-data/ts-1/data/info.pb
I20260924 21:08:50.302268 10133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:50.300401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:58065 (local address 127.0.98.62:33683)
0924 21:08:50.300459 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0924 21:08:50.300460 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 21:08:50.300647 (+ 187us) server_negotiation.cc:407] Connection header received
0924 21:08:50.301000 (+ 353us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:08:50.301001 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:08:50.301006 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:08:50.301052 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:08:50.302090 (+ 1038us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 21:08:50.302091 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 21:08:50.302092 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 21:08:50.302095 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0924 21:08:50.302108 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 21:08:50.302111 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 21:08:50.302156 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0924 21:08:50.302209 (+ 53us) server_negotiation.cc:299] Negotiation successful
0924 21:08:50.302220 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260924 21:08:50.302455 10415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:50.300385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33683 (local address 127.0.98.2:58065)
0924 21:08:50.300540 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0924 21:08:50.300573 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0924 21:08:50.300930 (+ 357us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 21:08:50.301091 (+ 161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 21:08:50.301094 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 21:08:50.301100 (+ 6us) client_negotiation.cc:189] Negotiated authn=SASL
0924 21:08:50.301971 (+ 871us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 21:08:50.301971 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0924 21:08:50.301994 (+ 23us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 21:08:50.302018 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 21:08:50.302154 (+ 136us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 21:08:50.302156 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 21:08:50.302158 (+ 2us) client_negotiation.cc:769] Sending connection context
0924 21:08:50.302205 (+ 47us) client_negotiation.cc:240] Negotiation successful
0924 21:08:50.302246 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":80,"threads_started":1}
I20260924 21:08:50.302850 10413 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33683
I20260924 21:08:50.302914 10413 heartbeater.cc:464] Registering TS with master...
I20260924 21:08:50.302999 10413 heartbeater.cc:511] Master 127.0.98.62:33683 requested a full tablet report, sending...
I20260924 21:08:50.303258 10094 ts_manager.cc:194] Registered new tserver with Master: d672fdfff72947dc8ccb2d2fbaf7ac1e (127.0.98.2:41141)
I20260924 21:08:50.309774 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAzNNjS/build/release/bin/kudu as pid 10285
I20260924 21:08:50.309893 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790284076800039-392-0/minicluster-data/ts-1/wal/instance
I20260924 21:08:50.310065 392 external_mini_cluster.cc:1428] Setting key 50502d025fe0ef47bc379c7ab804e178
I20260924 21:08:50.310842 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAzNNjS/build/release/bin/kudu
/tmp/dist-test-taskAzNNjS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790284076800039-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790284076800039-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790284076800039-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790284076800039-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33683
--builtin_ntp_servers=127.0.98.20:37441
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260924 21:08:50.383911 10416 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 21:08:50.384066 10416 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 21:08:50.384084 10416 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 21:08:50.385041 10416 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 21:08:50.385074 10416 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 21:08:50.385118 10416 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 21:08:50.386447 10416 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37441
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790284076800039-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790284076800039-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790284076800039-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33683
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10416
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790284076800039-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 19:49:06 UTC on ccycloud.quasar-vhqjoc.root.comops.site
build id 4670864
I20260924 21:08:50.386672 10416 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 21:08:50.386912 10416 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 21:08:50.391580 10422 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 21:08:50.391608 10421 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 21:08:50.391608 10424 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 21:08:50.391769 10416 server_base.cc:1034] running on GCE node
I20260924 21:08:50.391947 10416 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 21:08:50.392119 10416 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 21:08:50.393242 10416 hybrid_clock.cc:648] HybridClock initialized: now 1790284130393232 us; error 26 us; skew 500 ppm
I20260924 21:08:50.394415 10416 webserver.cc:460] Webserver started at http://127.0.98.3:41865/ using document root <none> and password file <none>
I20260924 21:08:50.394645 10416 fs_manager.cc:359] Metadata directory not provided
I20260924 21:08:50.394688 10416 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 21:08:50.394817 10416 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 21:08:50.395699 10416 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790284076800039-392-0/minicluster-data/ts-2/data/instance:
uuid: "30540e23eeb54906b286f12451f5e196"
format_stamp: "Formatted at 2026-09-24 21:08:50 on dist-test-slave-c7tk"
server_key: "ce92a163fba9db27de7633d0d71cfa1d"
server_key_iv: "4812948dd72167628c1f64aa5c050332"
server_key_version: "encryptionkey@0"
I20260924 21:08:50.396004 10416 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790284076800039-392-0/minicluster-data/ts-2/wal/instance:
uuid: "30540e23eeb54906b286f12451f5e196"
format_stamp: "Formatted at 2026-09-24 21:08:50 on dist-test-slave-c7tk"
server_key: "ce92a163fba9db27de7633d0d71cfa1d"
server_key_iv: "4812948dd72167628c1f64aa5c050332"
server_key_version: "encryptionkey@0"
I20260924 21:08:50.397150 10416 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260924 21:08:50.397712 10430 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 21:08:50.397861 10416 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 21:08:50.397935 10416 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790284076800039-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790284076800039-392-0/minicluster-data/ts-2/wal
uuid: "30540e23eeb54906b286f12451f5e196"
format_stamp: "Formatted at 2026-09-24 21:08:50 on dist-test-slave-c7tk"
server_key: "ce92a163fba9db27de7633d0d71cfa1d"
server_key_iv: "4812948dd72167628c1f64aa5c050332"
server_key_version: "encryptionkey@0"
I20260924 21:08:50.397986 10416 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790284076800039-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790284076800039-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790284076800039-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 21:08:50.400753 10416 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 21:08:50.400918 10416 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 21:08:50.400995 10416 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 21:08:50.401121 10416 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 21:08:50.401372 10416 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 21:08:50.401399 10416 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 21:08:50.401418 10416 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 21:08:50.401430 10416 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 21:08:50.406221 10416 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:41447
I20260924 21:08:50.406286 10543 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:41447 every 8 connection(s)
I20260924 21:08:50.406561 10416 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790284076800039-392-0/minicluster-data/ts-2/data/info.pb
I20260924 21:08:50.408896 10133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:50.407200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:40685 (local address 127.0.98.62:33683)
0924 21:08:50.407257 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0924 21:08:50.407258 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 21:08:50.407457 (+ 199us) server_negotiation.cc:407] Connection header received
0924 21:08:50.407793 (+ 336us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:08:50.407794 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:08:50.407797 (+ 3us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:08:50.407856 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:08:50.408728 (+ 872us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 21:08:50.408729 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 21:08:50.408729 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 21:08:50.408732 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0924 21:08:50.408751 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 21:08:50.408754 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 21:08:50.408797 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0924 21:08:50.408844 (+ 47us) server_negotiation.cc:299] Negotiation successful
0924 21:08:50.408853 (+ 9us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260924 21:08:50.409034 10546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:50.407182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33683 (local address 127.0.98.3:40685)
0924 21:08:50.407347 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0924 21:08:50.407379 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0924 21:08:50.407724 (+ 345us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 21:08:50.407893 (+ 169us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 21:08:50.407914 (+ 21us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 21:08:50.407921 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0924 21:08:50.408620 (+ 699us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 21:08:50.408621 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0924 21:08:50.408642 (+ 21us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 21:08:50.408662 (+ 20us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 21:08:50.408798 (+ 136us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 21:08:50.408799 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 21:08:50.408801 (+ 2us) client_negotiation.cc:769] Sending connection context
0924 21:08:50.408843 (+ 42us) client_negotiation.cc:240] Negotiation successful
0924 21:08:50.408878 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":79,"threads_started":1}
I20260924 21:08:50.409435 10544 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33683
I20260924 21:08:50.409507 10544 heartbeater.cc:464] Registering TS with master...
I20260924 21:08:50.409592 10544 heartbeater.cc:511] Master 127.0.98.62:33683 requested a full tablet report, sending...
I20260924 21:08:50.409878 10094 ts_manager.cc:194] Registered new tserver with Master: 30540e23eeb54906b286f12451f5e196 (127.0.98.3:41447)
I20260924 21:08:50.414606 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAzNNjS/build/release/bin/kudu as pid 10416
I20260924 21:08:50.414690 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790284076800039-392-0/minicluster-data/ts-2/wal/instance
I20260924 21:08:50.414885 392 external_mini_cluster.cc:1428] Setting key e4b88b49d183f10df45c19fafd36d037
I20260924 21:08:50.415484 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260924 21:08:50.416996 10133 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 21:08:50.416580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60080 (local address 127.0.98.62:33683)
0924 21:08:50.416636 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0924 21:08:50.416637 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 21:08:50.416721 (+ 84us) server_negotiation.cc:407] Connection header received
0924 21:08:50.416787 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:08:50.416788 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:08:50.416791 (+ 3us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:08:50.416838 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:08:50.416941 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60080: BlockingRecv error: recv got EOF from 127.0.0.1:60080 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
I20260924 21:08:50.417070 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAzNNjS/build/release/bin/kudu with pid 10154
I20260924 21:08:50.420949 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAzNNjS/build/release/bin/kudu with pid 10285
I20260924 21:08:50.425633 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAzNNjS/build/release/bin/kudu with pid 10416
I20260924 21:08:50.429826 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAzNNjS/build/release/bin/kudu with pid 10063
2026-09-24T21:08:50Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (484 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790284076800039-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 21:08:50 dist-test-slave-c7tk krb5kdc[10554](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 21:08:50 dist-test-slave-c7tk krb5kdc[10554](info): set up 2 sockets
Sep 24 21:08:50 dist-test-slave-c7tk krb5kdc[10554](info): commencing operation
krb5kdc: starting...
W20260924 21:08:51.350221 392 mini_kdc.cc:121] Time spent starting KDC: real 0.902s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790284076800039-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790284076800039-392-0/krb5kdc/test-user.keytab.
Sep 24 21:08:51 dist-test-slave-c7tk krb5kdc[10554](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284131, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T21:08:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T21:08:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790284076800039-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790284076800039-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 21:08:51.450297 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAzNNjS/build/release/bin/kudu
/tmp/dist-test-taskAzNNjS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790284076800039-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790284076800039-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790284076800039-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790284076800039-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37805
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790284076800039-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42001
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37805
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790284076800039-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790284076800039-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790284076800039-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 21:08:51.519811 10567 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 21:08:51.519999 10567 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 21:08:51.520028 10567 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 21:08:51.521046 10567 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 21:08:51.521107 10567 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 21:08:51.521128 10567 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 21:08:51.521155 10567 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 21:08:51.521181 10567 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 21:08:51.522639 10567 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42001
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790284076800039-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790284076800039-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37805
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790284076800039-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37805
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790284076800039-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10567
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790284076800039-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 19:49:06 UTC on ccycloud.quasar-vhqjoc.root.comops.site
build id 4670864
I20260924 21:08:51.522876 10567 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 21:08:51.523066 10567 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 21:08:51.527295 10575 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 21:08:51.527330 10572 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 21:08:51.527318 10573 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 21:08:51.527467 10567 server_base.cc:1034] running on GCE node
I20260924 21:08:51.527741 10567 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 21:08:51.528029 10567 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 21:08:51.529188 10567 hybrid_clock.cc:648] HybridClock initialized: now 1790284131529176 us; error 29 us; skew 500 ppm
Sep 24 21:08:51 dist-test-slave-c7tk krb5kdc[10554](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284131, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 21:08:51.531447 10567 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 21:08:51.531792 10567 webserver.cc:460] Webserver started at http://127.0.98.62:36591/ using document root <none> and password file <none>
I20260924 21:08:51.531952 10567 fs_manager.cc:359] Metadata directory not provided
I20260924 21:08:51.531991 10567 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 21:08:51.532073 10567 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 21:08:51.532948 10567 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790284076800039-392-0/minicluster-data/master-0/data/instance:
uuid: "b8f951d2bbf64758834562fd3a76dee9"
format_stamp: "Formatted at 2026-09-24 21:08:51 on dist-test-slave-c7tk"
server_key: "6e387b61b4ed111a3a62e3d63ac30345"
server_key_iv: "0fca8c3b34d10893d627132238640fc2"
server_key_version: "encryptionkey@0"
I20260924 21:08:51.533201 10567 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790284076800039-392-0/minicluster-data/master-0/wal/instance:
uuid: "b8f951d2bbf64758834562fd3a76dee9"
format_stamp: "Formatted at 2026-09-24 21:08:51 on dist-test-slave-c7tk"
server_key: "6e387b61b4ed111a3a62e3d63ac30345"
server_key_iv: "0fca8c3b34d10893d627132238640fc2"
server_key_version: "encryptionkey@0"
I20260924 21:08:51.534490 10567 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260924 21:08:51.535106 10582 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 21:08:51.535266 10567 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 21:08:51.535336 10567 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790284076800039-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790284076800039-392-0/minicluster-data/master-0/wal
uuid: "b8f951d2bbf64758834562fd3a76dee9"
format_stamp: "Formatted at 2026-09-24 21:08:51 on dist-test-slave-c7tk"
server_key: "6e387b61b4ed111a3a62e3d63ac30345"
server_key_iv: "0fca8c3b34d10893d627132238640fc2"
server_key_version: "encryptionkey@0"
I20260924 21:08:51.535399 10567 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790284076800039-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790284076800039-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790284076800039-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 21:08:51.551357 10567 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 21:08:51.552932 10567 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 21:08:51.553015 10567 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 21:08:51.556526 10567 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37805
I20260924 21:08:51.556571 10634 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37805 every 8 connection(s)
I20260924 21:08:51.556823 10567 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790284076800039-392-0/minicluster-data/master-0/data/info.pb
I20260924 21:08:51.557626 10635 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 21:08:51.559479 10635 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b8f951d2bbf64758834562fd3a76dee9: Bootstrap starting.
I20260924 21:08:51.559959 10635 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b8f951d2bbf64758834562fd3a76dee9: Neither blocks nor log segments found. Creating new log.
I20260924 21:08:51.560209 10635 log.cc:824] T 00000000000000000000000000000000 P b8f951d2bbf64758834562fd3a76dee9: Log is configured to *not* fsync() on all Append() calls
I20260924 21:08:51.560786 10635 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b8f951d2bbf64758834562fd3a76dee9: No bootstrap required, opened a new log
I20260924 21:08:51.562254 10635 raft_consensus.cc:348] T 00000000000000000000000000000000 P b8f951d2bbf64758834562fd3a76dee9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8f951d2bbf64758834562fd3a76dee9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37805 } }
I20260924 21:08:51.562398 10635 raft_consensus.cc:374] T 00000000000000000000000000000000 P b8f951d2bbf64758834562fd3a76dee9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 21:08:51.562445 10635 raft_consensus.cc:729] T 00000000000000000000000000000000 P b8f951d2bbf64758834562fd3a76dee9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b8f951d2bbf64758834562fd3a76dee9, State: Initialized, Role: FOLLOWER
I20260924 21:08:51.562599 10635 consensus_queue.cc:260] T 00000000000000000000000000000000 P b8f951d2bbf64758834562fd3a76dee9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8f951d2bbf64758834562fd3a76dee9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37805 } }
I20260924 21:08:51.562668 10635 raft_consensus.cc:388] T 00000000000000000000000000000000 P b8f951d2bbf64758834562fd3a76dee9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 21:08:51.562703 10635 raft_consensus.cc:482] T 00000000000000000000000000000000 P b8f951d2bbf64758834562fd3a76dee9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 21:08:51.562743 10635 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b8f951d2bbf64758834562fd3a76dee9 [term 0 FOLLOWER]: Advancing to term 1
I20260924 21:08:51.563436 10635 raft_consensus.cc:504] T 00000000000000000000000000000000 P b8f951d2bbf64758834562fd3a76dee9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8f951d2bbf64758834562fd3a76dee9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37805 } }
I20260924 21:08:51.563539 10635 leader_election.cc:302] T 00000000000000000000000000000000 P b8f951d2bbf64758834562fd3a76dee9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b8f951d2bbf64758834562fd3a76dee9; no voters:
I20260924 21:08:51.563696 10635 leader_election.cc:288] T 00000000000000000000000000000000 P b8f951d2bbf64758834562fd3a76dee9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 21:08:51.563737 10638 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b8f951d2bbf64758834562fd3a76dee9 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 21:08:51.563911 10635 sys_catalog.cc:564] T 00000000000000000000000000000000 P b8f951d2bbf64758834562fd3a76dee9 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 21:08:51.563925 10638 raft_consensus.cc:686] T 00000000000000000000000000000000 P b8f951d2bbf64758834562fd3a76dee9 [term 1 LEADER]: Becoming Leader. State: Replica: b8f951d2bbf64758834562fd3a76dee9, State: Running, Role: LEADER
I20260924 21:08:51.564111 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAzNNjS/build/release/bin/kudu as pid 10567
I20260924 21:08:51.564137 10638 consensus_queue.cc:237] T 00000000000000000000000000000000 P b8f951d2bbf64758834562fd3a76dee9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8f951d2bbf64758834562fd3a76dee9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37805 } }
I20260924 21:08:51.564213 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790284076800039-392-0/minicluster-data/master-0/wal/instance
I20260924 21:08:51.564393 392 external_mini_cluster.cc:1428] Setting key 4412514b9ec73b301048c9fc10e9296f
I20260924 21:08:51.564430 10639 sys_catalog.cc:455] T 00000000000000000000000000000000 P b8f951d2bbf64758834562fd3a76dee9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b8f951d2bbf64758834562fd3a76dee9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8f951d2bbf64758834562fd3a76dee9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37805 } } }
I20260924 21:08:51.564484 10639 sys_catalog.cc:458] T 00000000000000000000000000000000 P b8f951d2bbf64758834562fd3a76dee9 [sys.catalog]: This master's current role is: LEADER
I20260924 21:08:51.564615 10640 sys_catalog.cc:455] T 00000000000000000000000000000000 P b8f951d2bbf64758834562fd3a76dee9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b8f951d2bbf64758834562fd3a76dee9. Latest consensus state: current_term: 1 leader_uuid: "b8f951d2bbf64758834562fd3a76dee9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8f951d2bbf64758834562fd3a76dee9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37805 } } }
I20260924 21:08:51.564667 10640 sys_catalog.cc:458] T 00000000000000000000000000000000 P b8f951d2bbf64758834562fd3a76dee9 [sys.catalog]: This master's current role is: LEADER
I20260924 21:08:51.564824 10646 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 21:08:51.565313 10646 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 21:08:51.566604 10646 catalog_manager.cc:1347] Generated new cluster ID: 1dc621151f7d48ea9ed3045f898c914e
I20260924 21:08:51.566650 10646 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 24 21:08:51 dist-test-slave-c7tk krb5kdc[10554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284131, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 21:08:51.573606 10648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:51.564786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33688 (local address 127.0.98.62:37805)
0924 21:08:51.565049 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0924 21:08:51.565068 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0924 21:08:51.565742 (+ 674us) server_negotiation.cc:407] Connection header received
0924 21:08:51.565899 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:08:51.565905 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:08:51.565956 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:08:51.566083 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:08:51.566712 (+ 629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:51.567467 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:51.568158 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:51.568547 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:51.568655 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:51.570643 (+ 1988us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 21:08:51.570647 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 21:08:51.570647 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 21:08:51.570656 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 21:08:51.572174 (+ 1518us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:08:51.572567 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:08:51.572569 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:08:51.572570 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:08:51.572663 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:08:51.572855 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:08:51.572856 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:08:51.572856 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:08:51.573053 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 21:08:51.573193 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0924 21:08:51.573292 (+ 99us) server_negotiation.cc:299] Negotiation successful
0924 21:08:51.573407 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":221,"thread_start_us":64,"threads_started":1}
I20260924 21:08:51.577735 10646 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 21:08:51.578176 10646 catalog_manager.cc:1504] Loading token signing keys...
I20260924 21:08:51.580583 10646 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b8f951d2bbf64758834562fd3a76dee9: Generated new TSK 0
I20260924 21:08:51.580737 10646 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790284076800039-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790284076800039-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260924 21:08:51.610440 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAzNNjS/build/release/bin/kudu
/tmp/dist-test-taskAzNNjS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790284076800039-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790284076800039-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790284076800039-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790284076800039-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37805
--keytab_file=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790284076800039-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42001
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790284076800039-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790284076800039-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790284076800039-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 21:08:51.692046 10661 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 21:08:51.692229 10661 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 21:08:51.692257 10661 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 21:08:51.693257 10661 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 21:08:51.693292 10661 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 21:08:51.693315 10661 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 21:08:51.694656 10661 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42001
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790284076800039-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790284076800039-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790284076800039-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790284076800039-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37805
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10661
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790284076800039-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 19:49:06 UTC on ccycloud.quasar-vhqjoc.root.comops.site
build id 4670864
I20260924 21:08:51.694859 10661 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 21:08:51.695083 10661 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 21:08:51.700116 10667 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 21:08:51.700201 10661 server_base.cc:1034] running on GCE node
W20260924 21:08:51.700307 10666 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 21:08:51.700322 10669 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 21:08:51.700522 10661 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 21:08:51.700757 10661 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 21:08:51.701908 10661 hybrid_clock.cc:648] HybridClock initialized: now 1790284131701897 us; error 27 us; skew 500 ppm
Sep 24 21:08:51 dist-test-slave-c7tk krb5kdc[10554](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284131, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 21:08:51.703994 10661 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 21:08:51.704355 10661 webserver.cc:460] Webserver started at http://127.0.98.1:38273/ using document root <none> and password file <none>
I20260924 21:08:51.704543 10661 fs_manager.cc:359] Metadata directory not provided
I20260924 21:08:51.704602 10661 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 21:08:51.704715 10661 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 21:08:51.705550 10661 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790284076800039-392-0/minicluster-data/ts-0/data/instance:
uuid: "5bba75bcf25b4e7ca1c983368f8c1d70"
format_stamp: "Formatted at 2026-09-24 21:08:51 on dist-test-slave-c7tk"
server_key: "26b13bd0192769f12134b51b4f5b5f31"
server_key_iv: "f490ccb5112517276e4ed23cc149a785"
server_key_version: "encryptionkey@0"
I20260924 21:08:51.705915 10661 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790284076800039-392-0/minicluster-data/ts-0/wal/instance:
uuid: "5bba75bcf25b4e7ca1c983368f8c1d70"
format_stamp: "Formatted at 2026-09-24 21:08:51 on dist-test-slave-c7tk"
server_key: "26b13bd0192769f12134b51b4f5b5f31"
server_key_iv: "f490ccb5112517276e4ed23cc149a785"
server_key_version: "encryptionkey@0"
I20260924 21:08:51.707002 10661 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 21:08:51.707599 10676 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 21:08:51.707763 10661 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 21:08:51.707847 10661 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790284076800039-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790284076800039-392-0/minicluster-data/ts-0/wal
uuid: "5bba75bcf25b4e7ca1c983368f8c1d70"
format_stamp: "Formatted at 2026-09-24 21:08:51 on dist-test-slave-c7tk"
server_key: "26b13bd0192769f12134b51b4f5b5f31"
server_key_iv: "f490ccb5112517276e4ed23cc149a785"
server_key_version: "encryptionkey@0"
I20260924 21:08:51.707921 10661 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790284076800039-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790284076800039-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790284076800039-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 21:08:51.715915 10661 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 21:08:51.717538 10661 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 21:08:51.717689 10661 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 21:08:51.717940 10661 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 21:08:51.718235 10661 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 21:08:51.718292 10661 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 21:08:51.718327 10661 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 21:08:51.718353 10661 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 21:08:51.723441 10661 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:39831
I20260924 21:08:51.723495 10789 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:39831 every 8 connection(s)
I20260924 21:08:51.723752 10661 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790284076800039-392-0/minicluster-data/ts-0/data/info.pb
I20260924 21:08:51.724433 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAzNNjS/build/release/bin/kudu as pid 10661
I20260924 21:08:51.724557 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790284076800039-392-0/minicluster-data/ts-0/wal/instance
I20260924 21:08:51.724736 392 external_mini_cluster.cc:1428] Setting key 0c9b11fa330d43db0b1e9f316571751b
Sep 24 21:08:51 dist-test-slave-c7tk krb5kdc[10554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284131, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260924 21:08:51.733182 10648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:51.724346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:60267 (local address 127.0.98.62:37805)
0924 21:08:51.724407 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0924 21:08:51.724409 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 21:08:51.724579 (+ 170us) server_negotiation.cc:407] Connection header received
0924 21:08:51.725122 (+ 543us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:08:51.725123 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:08:51.725134 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:08:51.725191 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:08:51.726058 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:51.726744 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:51.728100 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:51.728295 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:51.728345 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:51.730431 (+ 2086us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 21:08:51.730433 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 21:08:51.730433 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 21:08:51.730441 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 21:08:51.731884 (+ 1443us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:08:51.732292 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:08:51.732293 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:08:51.732294 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:08:51.732352 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:08:51.732598 (+ 246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:08:51.732599 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:08:51.732600 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:08:51.732711 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 21:08:51.732780 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0924 21:08:51.733014 (+ 234us) server_negotiation.cc:299] Negotiation successful
0924 21:08:51.733095 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260924 21:08:51.733203 10792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:51.724315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37805 (local address 127.0.98.1:60267)
0924 21:08:51.724480 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0924 21:08:51.724513 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0924 21:08:51.725022 (+ 509us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 21:08:51.725260 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 21:08:51.725281 (+ 21us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 21:08:51.725777 (+ 496us) client_negotiation.cc:189] Negotiated authn=SASL
0924 21:08:51.725981 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:51.725985 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:51.726821 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:51.726823 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:51.728024 (+ 1201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:51.728027 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:51.728374 (+ 347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:51.728375 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:51.728507 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:51.728906 (+ 399us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 21:08:51.728916 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 21:08:51.730355 (+ 1439us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 21:08:51.731966 (+ 1611us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 21:08:51.731971 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 21:08:51.731973 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 21:08:51.732212 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 21:08:51.732426 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 21:08:51.732427 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 21:08:51.732427 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 21:08:51.732525 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 21:08:51.732775 (+ 250us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 21:08:51.732779 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 21:08:51.732865 (+ 86us) client_negotiation.cc:769] Sending connection context
0924 21:08:51.732985 (+ 120us) client_negotiation.cc:240] Negotiation successful
0924 21:08:51.733083 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":91,"threads_started":1}
I20260924 21:08:51.733675 10790 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37805
I20260924 21:08:51.733778 10790 heartbeater.cc:464] Registering TS with master...
I20260924 21:08:51.733953 10790 heartbeater.cc:511] Master 127.0.98.62:37805 requested a full tablet report, sending...
I20260924 21:08:51.734426 10599 ts_manager.cc:194] Registered new tserver with Master: 5bba75bcf25b4e7ca1c983368f8c1d70 (127.0.98.1:39831)
I20260924 21:08:51.734997 10599 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:60267
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790284076800039-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790284076800039-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260924 21:08:51.746539 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAzNNjS/build/release/bin/kudu
/tmp/dist-test-taskAzNNjS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790284076800039-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790284076800039-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790284076800039-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790284076800039-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37805
--keytab_file=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790284076800039-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42001
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790284076800039-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790284076800039-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790284076800039-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 21:08:51.815471 10795 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 21:08:51.815621 10795 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 21:08:51.815640 10795 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 21:08:51.816553 10795 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 21:08:51.816582 10795 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 21:08:51.816604 10795 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 21:08:51.817721 10795 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42001
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790284076800039-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790284076800039-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790284076800039-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790284076800039-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37805
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10795
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790284076800039-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 19:49:06 UTC on ccycloud.quasar-vhqjoc.root.comops.site
build id 4670864
I20260924 21:08:51.817919 10795 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 21:08:51.818056 10795 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 21:08:51.822459 10803 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 21:08:51.822475 10801 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 21:08:51.822459 10800 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 21:08:51.822803 10795 server_base.cc:1034] running on GCE node
I20260924 21:08:51.822970 10795 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 21:08:51.823143 10795 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 21:08:51.824268 10795 hybrid_clock.cc:648] HybridClock initialized: now 1790284131824258 us; error 26 us; skew 500 ppm
Sep 24 21:08:51 dist-test-slave-c7tk krb5kdc[10554](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284131, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 21:08:51.826711 10795 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 21:08:51.827054 10795 webserver.cc:460] Webserver started at http://127.0.98.2:43219/ using document root <none> and password file <none>
I20260924 21:08:51.827235 10795 fs_manager.cc:359] Metadata directory not provided
I20260924 21:08:51.827277 10795 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 21:08:51.827360 10795 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 21:08:51.828218 10795 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790284076800039-392-0/minicluster-data/ts-1/data/instance:
uuid: "1a3b362662f24aa5ba32e82b1e34ea5c"
format_stamp: "Formatted at 2026-09-24 21:08:51 on dist-test-slave-c7tk"
server_key: "ef3be82edfea0a82a219dcdd09f10c38"
server_key_iv: "314df8f28504f5c60366279d757f73f4"
server_key_version: "encryptionkey@0"
I20260924 21:08:51.828513 10795 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790284076800039-392-0/minicluster-data/ts-1/wal/instance:
uuid: "1a3b362662f24aa5ba32e82b1e34ea5c"
format_stamp: "Formatted at 2026-09-24 21:08:51 on dist-test-slave-c7tk"
server_key: "ef3be82edfea0a82a219dcdd09f10c38"
server_key_iv: "314df8f28504f5c60366279d757f73f4"
server_key_version: "encryptionkey@0"
I20260924 21:08:51.829703 10795 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260924 21:08:51.830394 10810 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 21:08:51.830530 10795 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 21:08:51.830600 10795 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790284076800039-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790284076800039-392-0/minicluster-data/ts-1/wal
uuid: "1a3b362662f24aa5ba32e82b1e34ea5c"
format_stamp: "Formatted at 2026-09-24 21:08:51 on dist-test-slave-c7tk"
server_key: "ef3be82edfea0a82a219dcdd09f10c38"
server_key_iv: "314df8f28504f5c60366279d757f73f4"
server_key_version: "encryptionkey@0"
I20260924 21:08:51.830646 10795 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790284076800039-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790284076800039-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790284076800039-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 21:08:51.851238 10795 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 21:08:51.853010 10795 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 21:08:51.853123 10795 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 21:08:51.853277 10795 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 21:08:51.853554 10795 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 21:08:51.853585 10795 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 21:08:51.853605 10795 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 21:08:51.853652 10795 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 21:08:51.859031 10795 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:39367
I20260924 21:08:51.859104 10923 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:39367 every 8 connection(s)
I20260924 21:08:51.859380 10795 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790284076800039-392-0/minicluster-data/ts-1/data/info.pb
I20260924 21:08:51.860000 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAzNNjS/build/release/bin/kudu as pid 10795
I20260924 21:08:51.860092 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790284076800039-392-0/minicluster-data/ts-1/wal/instance
I20260924 21:08:51.860301 392 external_mini_cluster.cc:1428] Setting key c511c204f5c020a88833f6f723db2612
Sep 24 21:08:51 dist-test-slave-c7tk krb5kdc[10554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284131, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260924 21:08:51.869192 10648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:51.860058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:38465 (local address 127.0.98.62:37805)
0924 21:08:51.860107 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0924 21:08:51.860108 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 21:08:51.860307 (+ 199us) server_negotiation.cc:407] Connection header received
0924 21:08:51.860876 (+ 569us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:08:51.860877 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:08:51.860888 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:08:51.860931 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:08:51.862208 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:51.862713 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:51.864127 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:51.864294 (+ 167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:51.864342 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:51.866698 (+ 2356us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 21:08:51.866699 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 21:08:51.866699 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 21:08:51.866706 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 21:08:51.867945 (+ 1239us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:08:51.868318 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:08:51.868319 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:08:51.868320 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:08:51.868371 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:08:51.868620 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:08:51.868621 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:08:51.868621 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:08:51.868721 (+ 100us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 21:08:51.868794 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0924 21:08:51.869042 (+ 248us) server_negotiation.cc:299] Negotiation successful
0924 21:08:51.869114 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260924 21:08:51.869354 10926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:51.859993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37805 (local address 127.0.98.2:38465)
0924 21:08:51.860174 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0924 21:08:51.860219 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0924 21:08:51.860755 (+ 536us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 21:08:51.861024 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 21:08:51.861029 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 21:08:51.861865 (+ 836us) client_negotiation.cc:189] Negotiated authn=SASL
0924 21:08:51.862125 (+ 260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:51.862131 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:51.862813 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:51.862814 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:51.864059 (+ 1245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:51.864062 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:51.864390 (+ 328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:51.864392 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:51.864524 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:51.864982 (+ 458us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 21:08:51.864991 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 21:08:51.866586 (+ 1595us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 21:08:51.868023 (+ 1437us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 21:08:51.868028 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 21:08:51.868031 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 21:08:51.868234 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 21:08:51.868442 (+ 208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 21:08:51.868442 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 21:08:51.868443 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 21:08:51.868569 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 21:08:51.868796 (+ 227us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 21:08:51.868802 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 21:08:51.868923 (+ 121us) client_negotiation.cc:769] Sending connection context
0924 21:08:51.869013 (+ 90us) client_negotiation.cc:240] Negotiation successful
0924 21:08:51.869144 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":70,"threads_started":1}
I20260924 21:08:51.869755 10924 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37805
I20260924 21:08:51.869872 10924 heartbeater.cc:464] Registering TS with master...
I20260924 21:08:51.869997 10924 heartbeater.cc:511] Master 127.0.98.62:37805 requested a full tablet report, sending...
I20260924 21:08:51.870294 10599 ts_manager.cc:194] Registered new tserver with Master: 1a3b362662f24aa5ba32e82b1e34ea5c (127.0.98.2:39367)
I20260924 21:08:51.870574 10599 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:38465
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790284076800039-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790284076800039-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260924 21:08:51.882634 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAzNNjS/build/release/bin/kudu
/tmp/dist-test-taskAzNNjS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790284076800039-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790284076800039-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790284076800039-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790284076800039-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37805
--keytab_file=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790284076800039-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42001
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790284076800039-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790284076800039-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790284076800039-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 21:08:51.952790 10929 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 21:08:51.952940 10929 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 21:08:51.952960 10929 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 21:08:51.953930 10929 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 21:08:51.953964 10929 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 21:08:51.954007 10929 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 21:08:51.955193 10929 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42001
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790284076800039-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790284076800039-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790284076800039-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790284076800039-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37805
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10929
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790284076800039-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 19:49:06 UTC on ccycloud.quasar-vhqjoc.root.comops.site
build id 4670864
I20260924 21:08:51.955396 10929 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 21:08:51.955582 10929 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 21:08:51.959827 10934 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 21:08:51.959827 10937 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 21:08:51.959843 10935 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 21:08:51.960263 10929 server_base.cc:1034] running on GCE node
I20260924 21:08:51.960451 10929 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 21:08:51.960645 10929 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 21:08:51.961826 10929 hybrid_clock.cc:648] HybridClock initialized: now 1790284131961777 us; error 30 us; skew 500 ppm
Sep 24 21:08:51 dist-test-slave-c7tk krb5kdc[10554](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284131, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 21:08:51.964159 10929 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 21:08:51.964543 10929 webserver.cc:460] Webserver started at http://127.0.98.3:44839/ using document root <none> and password file <none>
I20260924 21:08:51.964752 10929 fs_manager.cc:359] Metadata directory not provided
I20260924 21:08:51.964816 10929 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 21:08:51.964924 10929 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 21:08:51.965770 10929 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790284076800039-392-0/minicluster-data/ts-2/data/instance:
uuid: "381653909834491e8889720c91a2985a"
format_stamp: "Formatted at 2026-09-24 21:08:51 on dist-test-slave-c7tk"
server_key: "9f725055d43eb2f8e09e9f8404baa192"
server_key_iv: "62eb996ca28edd71dce73c992f8714d8"
server_key_version: "encryptionkey@0"
I20260924 21:08:51.966114 10929 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790284076800039-392-0/minicluster-data/ts-2/wal/instance:
uuid: "381653909834491e8889720c91a2985a"
format_stamp: "Formatted at 2026-09-24 21:08:51 on dist-test-slave-c7tk"
server_key: "9f725055d43eb2f8e09e9f8404baa192"
server_key_iv: "62eb996ca28edd71dce73c992f8714d8"
server_key_version: "encryptionkey@0"
I20260924 21:08:51.967173 10929 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 21:08:51.967782 10944 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 21:08:51.967937 10929 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 21:08:51.968019 10929 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790284076800039-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790284076800039-392-0/minicluster-data/ts-2/wal
uuid: "381653909834491e8889720c91a2985a"
format_stamp: "Formatted at 2026-09-24 21:08:51 on dist-test-slave-c7tk"
server_key: "9f725055d43eb2f8e09e9f8404baa192"
server_key_iv: "62eb996ca28edd71dce73c992f8714d8"
server_key_version: "encryptionkey@0"
I20260924 21:08:51.968091 10929 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790284076800039-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790284076800039-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790284076800039-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 21:08:51.989010 10929 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 21:08:51.991016 10929 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 21:08:51.991168 10929 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 21:08:51.991376 10929 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 21:08:51.991658 10929 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 21:08:51.991715 10929 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 21:08:51.991747 10929 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 21:08:51.991778 10929 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 21:08:51.997182 10929 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:33987
I20260924 21:08:51.997233 11057 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:33987 every 8 connection(s)
I20260924 21:08:51.997504 10929 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790284076800039-392-0/minicluster-data/ts-2/data/info.pb
Sep 24 21:08:51 dist-test-slave-c7tk krb5kdc[10554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284131, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 21:08:52.004655 10648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:51.998119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:51833 (local address 127.0.98.62:37805)
0924 21:08:51.998178 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0924 21:08:51.998180 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 21:08:51.998337 (+ 157us) server_negotiation.cc:407] Connection header received
0924 21:08:51.998691 (+ 354us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:08:51.998692 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:08:51.998701 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:08:51.998758 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:08:51.999459 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:51.999876 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:52.000752 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:52.000907 (+ 155us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:52.000952 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:52.002816 (+ 1864us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 21:08:52.002818 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 21:08:52.002818 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 21:08:52.002823 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 21:08:52.003785 (+ 962us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:08:52.004079 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:08:52.004080 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:08:52.004080 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:08:52.004121 (+ 41us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:08:52.004273 (+ 152us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:08:52.004273 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:08:52.004274 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:08:52.004353 (+ 79us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 21:08:52.004394 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0924 21:08:52.004550 (+ 156us) server_negotiation.cc:299] Negotiation successful
0924 21:08:52.004603 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260924 21:08:52.004793 11060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:51.998094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37805 (local address 127.0.98.3:51833)
0924 21:08:51.998234 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0924 21:08:51.998266 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0924 21:08:51.998623 (+ 357us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 21:08:51.998803 (+ 180us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 21:08:51.998805 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 21:08:51.999245 (+ 440us) client_negotiation.cc:189] Negotiated authn=SASL
0924 21:08:51.999393 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:51.999396 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:51.999976 (+ 580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:51.999977 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:52.000673 (+ 696us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:52.000674 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:52.000968 (+ 294us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:52.000969 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:52.001056 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:52.001422 (+ 366us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 21:08:52.001429 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0924 21:08:52.002745 (+ 1316us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 21:08:52.003844 (+ 1099us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 21:08:52.003848 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 21:08:52.003850 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 21:08:52.004024 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 21:08:52.004159 (+ 135us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 21:08:52.004159 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 21:08:52.004160 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 21:08:52.004232 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 21:08:52.004385 (+ 153us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 21:08:52.004387 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 21:08:52.004466 (+ 79us) client_negotiation.cc:769] Sending connection context
0924 21:08:52.004526 (+ 60us) client_negotiation.cc:240] Negotiation successful
0924 21:08:52.004613 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":72,"threads_started":1}
I20260924 21:08:52.005173 11058 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37805
I20260924 21:08:52.005259 11058 heartbeater.cc:464] Registering TS with master...
I20260924 21:08:52.005406 11058 heartbeater.cc:511] Master 127.0.98.62:37805 requested a full tablet report, sending...
I20260924 21:08:52.005709 10599 ts_manager.cc:194] Registered new tserver with Master: 381653909834491e8889720c91a2985a (127.0.98.3:33987)
I20260924 21:08:52.006036 10599 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:51833
I20260924 21:08:52.006417 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAzNNjS/build/release/bin/kudu as pid 10929
I20260924 21:08:52.006502 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790284076800039-392-0/minicluster-data/ts-2/wal/instance
I20260924 21:08:52.006664 392 external_mini_cluster.cc:1428] Setting key b5587a7ffe1498d2cab4b5ae2e908bb8
I20260924 21:08:52.007339 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 21:08:52.013360 10648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:52.008418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33692 (local address 127.0.98.62:37805)
0924 21:08:52.008473 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0924 21:08:52.008475 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 21:08:52.008566 (+ 91us) server_negotiation.cc:407] Connection header received
0924 21:08:52.008606 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:08:52.008606 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:08:52.008630 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:08:52.008674 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:08:52.009159 (+ 485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:52.009550 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:52.010253 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:52.010396 (+ 143us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:52.010440 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:52.011266 (+ 826us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 21:08:52.011267 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 21:08:52.011268 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 21:08:52.011273 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 21:08:52.012649 (+ 1376us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:08:52.012913 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:08:52.012913 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:08:52.012914 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:08:52.012954 (+ 40us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:08:52.013058 (+ 104us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:08:52.013058 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:08:52.013059 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:08:52.013149 (+ 90us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 21:08:52.013214 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0924 21:08:52.013271 (+ 57us) server_negotiation.cc:299] Negotiation successful
0924 21:08:52.013315 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260924 21:08:52.014753 10599 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33692:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 21:08:52.016095 10599 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 21:08:52.021616 11067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:52.017961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39367 (local address 127.0.0.1:50470)
0924 21:08:52.018094 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0924 21:08:52.018102 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0924 21:08:52.018138 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 21:08:52.018675 (+ 537us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 21:08:52.018676 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 21:08:52.018677 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 21:08:52.018720 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:52.018722 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:52.019893 (+ 1171us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:52.019894 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:52.020561 (+ 667us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:52.020562 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:52.021303 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:52.021304 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:52.021391 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:52.021403 (+ 12us) client_negotiation.cc:769] Sending connection context
0924 21:08:52.021501 (+ 98us) client_negotiation.cc:240] Negotiation successful
0924 21:08:52.021551 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":78,"threads_started":1}
I20260924 21:08:52.021629 11071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:52.018321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50470 (local address 127.0.98.2:39367)
0924 21:08:52.018523 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0924 21:08:52.018526 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 21:08:52.018534 (+ 8us) server_negotiation.cc:407] Connection header received
0924 21:08:52.018565 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:08:52.018567 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:08:52.018582 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:08:52.018822 (+ 240us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 21:08:52.018886 (+ 64us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:52.019828 (+ 942us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:52.020657 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:52.021213 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:52.021288 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:52.021344 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0924 21:08:52.021517 (+ 173us) server_negotiation.cc:299] Negotiation successful
0924 21:08:52.021553 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":165,"thread_start_us":110,"threads_started":1}
I20260924 21:08:52.021885 11069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:52.018132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33987 (local address 127.0.0.1:42068)
0924 21:08:52.018226 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0924 21:08:52.018233 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0924 21:08:52.018265 (+ 32us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 21:08:52.018646 (+ 381us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 21:08:52.018646 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 21:08:52.018649 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 21:08:52.018706 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:52.018708 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:52.019825 (+ 1117us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:52.019825 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:52.020775 (+ 950us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:52.020776 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:52.021625 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:52.021626 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:52.021687 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:52.021691 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 21:08:52.021727 (+ 36us) client_negotiation.cc:240] Negotiation successful
0924 21:08:52.021771 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":82,"mutex_wait_us":3,"thread_start_us":64,"threads_started":1}
I20260924 21:08:52.022025 11066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:52.017866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39831 (local address 127.0.0.1:39366)
0924 21:08:52.017956 (+ 90us) negotiation.cc:107] Waiting for socket to connect
0924 21:08:52.017985 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0924 21:08:52.018056 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 21:08:52.018351 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 21:08:52.018354 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 21:08:52.018356 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 21:08:52.018436 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:52.018438 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:52.019606 (+ 1168us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:52.019607 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:52.020718 (+ 1111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:52.020721 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:52.021688 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:52.021689 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:52.021745 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:52.021748 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 21:08:52.021838 (+ 90us) client_negotiation.cc:240] Negotiation successful
0924 21:08:52.021874 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":73,"thread_start_us":53,"threads_started":1}
I20260924 21:08:52.022065 11068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:52.017968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39366 (local address 127.0.98.1:39831)
0924 21:08:52.018186 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0924 21:08:52.018190 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 21:08:52.018198 (+ 8us) server_negotiation.cc:407] Connection header received
0924 21:08:52.018245 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:08:52.018247 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:08:52.018261 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:08:52.018336 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 21:08:52.018534 (+ 198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:52.019494 (+ 960us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:52.020821 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:52.021514 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:52.021592 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:52.021644 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0924 21:08:52.021933 (+ 289us) server_negotiation.cc:299] Negotiation successful
0924 21:08:52.021982 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":188,"thread_start_us":73,"threads_started":1}
I20260924 21:08:52.022392 11070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:52.018203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42068 (local address 127.0.98.3:33987)
0924 21:08:52.018496 (+ 293us) server_negotiation.cc:206] Beginning negotiation
0924 21:08:52.018500 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 21:08:52.018508 (+ 8us) server_negotiation.cc:407] Connection header received
0924 21:08:52.018538 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:08:52.018540 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:08:52.018553 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:08:52.018632 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 21:08:52.018789 (+ 157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:52.019732 (+ 943us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:52.020859 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:52.021514 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:52.022175 (+ 661us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:52.022233 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0924 21:08:52.022282 (+ 49us) server_negotiation.cc:299] Negotiation successful
0924 21:08:52.022319 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":266,"thread_start_us":57,"threads_started":1}
I20260924 21:08:52.022694 10858 tablet_service.cc:1467] Processing CreateTablet for tablet 403a964f77564be5a322d6b78d01318d (DEFAULT_TABLE table=test-table [id=f19ef6c32df140d983c151a1cde690cb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 21:08:52.022846 10724 tablet_service.cc:1467] Processing CreateTablet for tablet 403a964f77564be5a322d6b78d01318d (DEFAULT_TABLE table=test-table [id=f19ef6c32df140d983c151a1cde690cb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 21:08:52.022980 10858 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 403a964f77564be5a322d6b78d01318d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 21:08:52.022934 10992 tablet_service.cc:1467] Processing CreateTablet for tablet 403a964f77564be5a322d6b78d01318d (DEFAULT_TABLE table=test-table [id=f19ef6c32df140d983c151a1cde690cb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 21:08:52.023166 10724 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 403a964f77564be5a322d6b78d01318d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 21:08:52.023165 10992 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 403a964f77564be5a322d6b78d01318d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 21:08:52.025102 11072 tablet_bootstrap.cc:492] T 403a964f77564be5a322d6b78d01318d P 1a3b362662f24aa5ba32e82b1e34ea5c: Bootstrap starting.
I20260924 21:08:52.025627 11072 tablet_bootstrap.cc:654] T 403a964f77564be5a322d6b78d01318d P 1a3b362662f24aa5ba32e82b1e34ea5c: Neither blocks nor log segments found. Creating new log.
I20260924 21:08:52.025856 11074 tablet_bootstrap.cc:492] T 403a964f77564be5a322d6b78d01318d P 5bba75bcf25b4e7ca1c983368f8c1d70: Bootstrap starting.
I20260924 21:08:52.026216 11072 log.cc:824] T 403a964f77564be5a322d6b78d01318d P 1a3b362662f24aa5ba32e82b1e34ea5c: Log is configured to *not* fsync() on all Append() calls
I20260924 21:08:52.026532 11074 tablet_bootstrap.cc:654] T 403a964f77564be5a322d6b78d01318d P 5bba75bcf25b4e7ca1c983368f8c1d70: Neither blocks nor log segments found. Creating new log.
I20260924 21:08:52.026829 11072 tablet_bootstrap.cc:492] T 403a964f77564be5a322d6b78d01318d P 1a3b362662f24aa5ba32e82b1e34ea5c: No bootstrap required, opened a new log
I20260924 21:08:52.026876 11072 ts_tablet_manager.cc:1397] T 403a964f77564be5a322d6b78d01318d P 1a3b362662f24aa5ba32e82b1e34ea5c: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 21:08:52.027007 11074 log.cc:824] T 403a964f77564be5a322d6b78d01318d P 5bba75bcf25b4e7ca1c983368f8c1d70: Log is configured to *not* fsync() on all Append() calls
I20260924 21:08:52.027315 11073 tablet_bootstrap.cc:492] T 403a964f77564be5a322d6b78d01318d P 381653909834491e8889720c91a2985a: Bootstrap starting.
I20260924 21:08:52.027666 11074 tablet_bootstrap.cc:492] T 403a964f77564be5a322d6b78d01318d P 5bba75bcf25b4e7ca1c983368f8c1d70: No bootstrap required, opened a new log
I20260924 21:08:52.027729 11074 ts_tablet_manager.cc:1397] T 403a964f77564be5a322d6b78d01318d P 5bba75bcf25b4e7ca1c983368f8c1d70: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 21:08:52.027834 11073 tablet_bootstrap.cc:654] T 403a964f77564be5a322d6b78d01318d P 381653909834491e8889720c91a2985a: Neither blocks nor log segments found. Creating new log.
I20260924 21:08:52.028246 11073 log.cc:824] T 403a964f77564be5a322d6b78d01318d P 381653909834491e8889720c91a2985a: Log is configured to *not* fsync() on all Append() calls
I20260924 21:08:52.028458 11072 raft_consensus.cc:348] T 403a964f77564be5a322d6b78d01318d P 1a3b362662f24aa5ba32e82b1e34ea5c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bba75bcf25b4e7ca1c983368f8c1d70" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39831 } } peers { permanent_uuid: "1a3b362662f24aa5ba32e82b1e34ea5c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39367 } } peers { permanent_uuid: "381653909834491e8889720c91a2985a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33987 } }
I20260924 21:08:52.028609 11072 raft_consensus.cc:374] T 403a964f77564be5a322d6b78d01318d P 1a3b362662f24aa5ba32e82b1e34ea5c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 21:08:52.028640 11072 raft_consensus.cc:729] T 403a964f77564be5a322d6b78d01318d P 1a3b362662f24aa5ba32e82b1e34ea5c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a3b362662f24aa5ba32e82b1e34ea5c, State: Initialized, Role: FOLLOWER
I20260924 21:08:52.028918 11073 tablet_bootstrap.cc:492] T 403a964f77564be5a322d6b78d01318d P 381653909834491e8889720c91a2985a: No bootstrap required, opened a new log
I20260924 21:08:52.028992 11073 ts_tablet_manager.cc:1397] T 403a964f77564be5a322d6b78d01318d P 381653909834491e8889720c91a2985a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 21:08:52.028987 11072 consensus_queue.cc:260] T 403a964f77564be5a322d6b78d01318d P 1a3b362662f24aa5ba32e82b1e34ea5c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bba75bcf25b4e7ca1c983368f8c1d70" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39831 } } peers { permanent_uuid: "1a3b362662f24aa5ba32e82b1e34ea5c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39367 } } peers { permanent_uuid: "381653909834491e8889720c91a2985a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33987 } }
I20260924 21:08:52.029243 11072 ts_tablet_manager.cc:1428] T 403a964f77564be5a322d6b78d01318d P 1a3b362662f24aa5ba32e82b1e34ea5c: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 21:08:52.029271 10924 heartbeater.cc:503] Master 127.0.98.62:37805 was elected leader, sending a full tablet report...
I20260924 21:08:52.029507 11074 raft_consensus.cc:348] T 403a964f77564be5a322d6b78d01318d P 5bba75bcf25b4e7ca1c983368f8c1d70 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bba75bcf25b4e7ca1c983368f8c1d70" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39831 } } peers { permanent_uuid: "1a3b362662f24aa5ba32e82b1e34ea5c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39367 } } peers { permanent_uuid: "381653909834491e8889720c91a2985a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33987 } }
I20260924 21:08:52.029655 11074 raft_consensus.cc:374] T 403a964f77564be5a322d6b78d01318d P 5bba75bcf25b4e7ca1c983368f8c1d70 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 21:08:52.029685 11074 raft_consensus.cc:729] T 403a964f77564be5a322d6b78d01318d P 5bba75bcf25b4e7ca1c983368f8c1d70 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5bba75bcf25b4e7ca1c983368f8c1d70, State: Initialized, Role: FOLLOWER
I20260924 21:08:52.029790 11074 consensus_queue.cc:260] T 403a964f77564be5a322d6b78d01318d P 5bba75bcf25b4e7ca1c983368f8c1d70 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bba75bcf25b4e7ca1c983368f8c1d70" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39831 } } peers { permanent_uuid: "1a3b362662f24aa5ba32e82b1e34ea5c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39367 } } peers { permanent_uuid: "381653909834491e8889720c91a2985a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33987 } }
I20260924 21:08:52.030054 11074 ts_tablet_manager.cc:1428] T 403a964f77564be5a322d6b78d01318d P 5bba75bcf25b4e7ca1c983368f8c1d70: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 21:08:52.030167 10790 heartbeater.cc:503] Master 127.0.98.62:37805 was elected leader, sending a full tablet report...
I20260924 21:08:52.030766 11073 raft_consensus.cc:348] T 403a964f77564be5a322d6b78d01318d P 381653909834491e8889720c91a2985a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bba75bcf25b4e7ca1c983368f8c1d70" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39831 } } peers { permanent_uuid: "1a3b362662f24aa5ba32e82b1e34ea5c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39367 } } peers { permanent_uuid: "381653909834491e8889720c91a2985a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33987 } }
I20260924 21:08:52.030906 11073 raft_consensus.cc:374] T 403a964f77564be5a322d6b78d01318d P 381653909834491e8889720c91a2985a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 21:08:52.030947 11073 raft_consensus.cc:729] T 403a964f77564be5a322d6b78d01318d P 381653909834491e8889720c91a2985a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 381653909834491e8889720c91a2985a, State: Initialized, Role: FOLLOWER
I20260924 21:08:52.031042 11073 consensus_queue.cc:260] T 403a964f77564be5a322d6b78d01318d P 381653909834491e8889720c91a2985a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bba75bcf25b4e7ca1c983368f8c1d70" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39831 } } peers { permanent_uuid: "1a3b362662f24aa5ba32e82b1e34ea5c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39367 } } peers { permanent_uuid: "381653909834491e8889720c91a2985a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33987 } }
I20260924 21:08:52.031291 11058 heartbeater.cc:503] Master 127.0.98.62:37805 was elected leader, sending a full tablet report...
I20260924 21:08:52.031311 11073 ts_tablet_manager.cc:1428] T 403a964f77564be5a322d6b78d01318d P 381653909834491e8889720c91a2985a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
W20260924 21:08:52.109899 10925 tablet.cc:2369] T 403a964f77564be5a322d6b78d01318d P 1a3b362662f24aa5ba32e82b1e34ea5c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 21:08:52.224426 10791 tablet.cc:2369] T 403a964f77564be5a322d6b78d01318d P 5bba75bcf25b4e7ca1c983368f8c1d70: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 21:08:52.247999 11059 tablet.cc:2369] T 403a964f77564be5a322d6b78d01318d P 381653909834491e8889720c91a2985a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 21:08:52.296573 11079 raft_consensus.cc:482] T 403a964f77564be5a322d6b78d01318d P 5bba75bcf25b4e7ca1c983368f8c1d70 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 21:08:52.296741 11079 raft_consensus.cc:504] T 403a964f77564be5a322d6b78d01318d P 5bba75bcf25b4e7ca1c983368f8c1d70 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bba75bcf25b4e7ca1c983368f8c1d70" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39831 } } peers { permanent_uuid: "1a3b362662f24aa5ba32e82b1e34ea5c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39367 } } peers { permanent_uuid: "381653909834491e8889720c91a2985a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33987 } }
I20260924 21:08:52.297071 11079 leader_election.cc:288] T 403a964f77564be5a322d6b78d01318d P 5bba75bcf25b4e7ca1c983368f8c1d70 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1a3b362662f24aa5ba32e82b1e34ea5c (127.0.98.2:39367), 381653909834491e8889720c91a2985a (127.0.98.3:33987)
I20260924 21:08:52.299619 11071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:52.297346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:47005 (local address 127.0.98.2:39367)
0924 21:08:52.297400 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0924 21:08:52.297402 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 21:08:52.297414 (+ 12us) server_negotiation.cc:407] Connection header received
0924 21:08:52.297525 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:08:52.297526 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:08:52.297541 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:08:52.297579 (+ 38us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 21:08:52.297761 (+ 182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:52.298180 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:52.298986 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:52.299301 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:52.299374 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:52.299382 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0924 21:08:52.299513 (+ 131us) server_negotiation.cc:299] Negotiation successful
0924 21:08:52.299562 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260924 21:08:52.299634 11081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:52.297253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39367 (local address 127.0.98.1:47005)
0924 21:08:52.297356 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0924 21:08:52.297366 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 21:08:52.297417 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 21:08:52.297596 (+ 179us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 21:08:52.297597 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 21:08:52.297599 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 21:08:52.297667 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:52.297669 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:52.298269 (+ 600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:52.298270 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:52.298903 (+ 633us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:52.298905 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:52.299374 (+ 469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:52.299375 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:52.299437 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:52.299440 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 21:08:52.299505 (+ 65us) client_negotiation.cc:240] Negotiation successful
0924 21:08:52.299558 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":87,"thread_start_us":62,"threads_started":1}
I20260924 21:08:52.299711 11070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:52.297488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:52687 (local address 127.0.98.3:33987)
0924 21:08:52.297538 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0924 21:08:52.297540 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 21:08:52.297548 (+ 8us) server_negotiation.cc:407] Connection header received
0924 21:08:52.297632 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:08:52.297633 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:08:52.297644 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:08:52.297707 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 21:08:52.297868 (+ 161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:52.298263 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:52.299069 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:52.299388 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:52.299458 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:52.299467 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0924 21:08:52.299622 (+ 155us) server_negotiation.cc:299] Negotiation successful
0924 21:08:52.299654 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260924 21:08:52.299738 11082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:52.297409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33987 (local address 127.0.98.1:52687)
0924 21:08:52.297491 (+ 82us) negotiation.cc:107] Waiting for socket to connect
0924 21:08:52.297501 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 21:08:52.297565 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 21:08:52.297712 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 21:08:52.297713 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 21:08:52.297714 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 21:08:52.297780 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:52.297781 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:52.298337 (+ 556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:52.298338 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:52.298987 (+ 649us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:08:52.298989 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:08:52.299469 (+ 480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:08:52.299471 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:08:52.299553 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:52.299556 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 21:08:52.299616 (+ 60us) client_negotiation.cc:240] Negotiation successful
0924 21:08:52.299653 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":69,"thread_start_us":35,"threads_started":1}
I20260924 21:08:52.299973 11012 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "403a964f77564be5a322d6b78d01318d" candidate_uuid: "5bba75bcf25b4e7ca1c983368f8c1d70" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "381653909834491e8889720c91a2985a" is_pre_election: true
I20260924 21:08:52.299980 10878 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "403a964f77564be5a322d6b78d01318d" candidate_uuid: "5bba75bcf25b4e7ca1c983368f8c1d70" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1a3b362662f24aa5ba32e82b1e34ea5c" is_pre_election: true
I20260924 21:08:52.300099 10878 raft_consensus.cc:2454] T 403a964f77564be5a322d6b78d01318d P 1a3b362662f24aa5ba32e82b1e34ea5c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5bba75bcf25b4e7ca1c983368f8c1d70 in term 0.
I20260924 21:08:52.300099 11012 raft_consensus.cc:2454] T 403a964f77564be5a322d6b78d01318d P 381653909834491e8889720c91a2985a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5bba75bcf25b4e7ca1c983368f8c1d70 in term 0.
I20260924 21:08:52.300349 10679 leader_election.cc:302] T 403a964f77564be5a322d6b78d01318d P 5bba75bcf25b4e7ca1c983368f8c1d70 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1a3b362662f24aa5ba32e82b1e34ea5c, 5bba75bcf25b4e7ca1c983368f8c1d70; no voters:
I20260924 21:08:52.300546 11079 raft_consensus.cc:2781] T 403a964f77564be5a322d6b78d01318d P 5bba75bcf25b4e7ca1c983368f8c1d70 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 21:08:52.300670 11079 raft_consensus.cc:482] T 403a964f77564be5a322d6b78d01318d P 5bba75bcf25b4e7ca1c983368f8c1d70 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 21:08:52.300695 11079 raft_consensus.cc:3037] T 403a964f77564be5a322d6b78d01318d P 5bba75bcf25b4e7ca1c983368f8c1d70 [term 0 FOLLOWER]: Advancing to term 1
I20260924 21:08:52.301447 11079 raft_consensus.cc:504] T 403a964f77564be5a322d6b78d01318d P 5bba75bcf25b4e7ca1c983368f8c1d70 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bba75bcf25b4e7ca1c983368f8c1d70" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39831 } } peers { permanent_uuid: "1a3b362662f24aa5ba32e82b1e34ea5c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39367 } } peers { permanent_uuid: "381653909834491e8889720c91a2985a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33987 } }
I20260924 21:08:52.301538 11079 leader_election.cc:288] T 403a964f77564be5a322d6b78d01318d P 5bba75bcf25b4e7ca1c983368f8c1d70 [CANDIDATE]: Term 1 election: Requested vote from peers 1a3b362662f24aa5ba32e82b1e34ea5c (127.0.98.2:39367), 381653909834491e8889720c91a2985a (127.0.98.3:33987)
I20260924 21:08:52.301680 10878 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "403a964f77564be5a322d6b78d01318d" candidate_uuid: "5bba75bcf25b4e7ca1c983368f8c1d70" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1a3b362662f24aa5ba32e82b1e34ea5c"
I20260924 21:08:52.301720 11012 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "403a964f77564be5a322d6b78d01318d" candidate_uuid: "5bba75bcf25b4e7ca1c983368f8c1d70" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "381653909834491e8889720c91a2985a"
I20260924 21:08:52.301777 10878 raft_consensus.cc:3037] T 403a964f77564be5a322d6b78d01318d P 1a3b362662f24aa5ba32e82b1e34ea5c [term 0 FOLLOWER]: Advancing to term 1
I20260924 21:08:52.301787 11012 raft_consensus.cc:3037] T 403a964f77564be5a322d6b78d01318d P 381653909834491e8889720c91a2985a [term 0 FOLLOWER]: Advancing to term 1
I20260924 21:08:52.302691 10878 raft_consensus.cc:2454] T 403a964f77564be5a322d6b78d01318d P 1a3b362662f24aa5ba32e82b1e34ea5c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5bba75bcf25b4e7ca1c983368f8c1d70 in term 1.
I20260924 21:08:52.302691 11012 raft_consensus.cc:2454] T 403a964f77564be5a322d6b78d01318d P 381653909834491e8889720c91a2985a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5bba75bcf25b4e7ca1c983368f8c1d70 in term 1.
I20260924 21:08:52.302835 10679 leader_election.cc:302] T 403a964f77564be5a322d6b78d01318d P 5bba75bcf25b4e7ca1c983368f8c1d70 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1a3b362662f24aa5ba32e82b1e34ea5c, 5bba75bcf25b4e7ca1c983368f8c1d70; no voters:
I20260924 21:08:52.302909 11079 raft_consensus.cc:2781] T 403a964f77564be5a322d6b78d01318d P 5bba75bcf25b4e7ca1c983368f8c1d70 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 21:08:52.303038 11079 raft_consensus.cc:686] T 403a964f77564be5a322d6b78d01318d P 5bba75bcf25b4e7ca1c983368f8c1d70 [term 1 LEADER]: Becoming Leader. State: Replica: 5bba75bcf25b4e7ca1c983368f8c1d70, State: Running, Role: LEADER
I20260924 21:08:52.303135 11079 consensus_queue.cc:237] T 403a964f77564be5a322d6b78d01318d P 5bba75bcf25b4e7ca1c983368f8c1d70 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bba75bcf25b4e7ca1c983368f8c1d70" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39831 } } peers { permanent_uuid: "1a3b362662f24aa5ba32e82b1e34ea5c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39367 } } peers { permanent_uuid: "381653909834491e8889720c91a2985a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33987 } }
I20260924 21:08:52.303916 10598 catalog_manager.cc:5495] T 403a964f77564be5a322d6b78d01318d P 5bba75bcf25b4e7ca1c983368f8c1d70 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5bba75bcf25b4e7ca1c983368f8c1d70 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "5bba75bcf25b4e7ca1c983368f8c1d70" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bba75bcf25b4e7ca1c983368f8c1d70" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39831 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1a3b362662f24aa5ba32e82b1e34ea5c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39367 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "381653909834491e8889720c91a2985a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33987 } health_report { overall_health: UNKNOWN } } }
I20260924 21:08:52.375201 11068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:52.372987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39372 (local address 127.0.98.1:39831)
0924 21:08:52.373043 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0924 21:08:52.373044 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 21:08:52.373052 (+ 8us) server_negotiation.cc:407] Connection header received
0924 21:08:52.373094 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:08:52.373094 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:08:52.373114 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:08:52.373168 (+ 54us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 21:08:52.373289 (+ 121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:52.373757 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:52.374475 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:08:52.374664 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:08:52.374711 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:08:52.374894 (+ 183us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 21:08:52.374985 (+ 91us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 21:08:52.375038 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0924 21:08:52.375086 (+ 48us) server_negotiation.cc:299] Negotiation successful
0924 21:08:52.375100 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260924 21:08:52.376806 11012 raft_consensus.cc:1261] T 403a964f77564be5a322d6b78d01318d P 381653909834491e8889720c91a2985a [term 1 FOLLOWER]: Refusing update from remote peer 5bba75bcf25b4e7ca1c983368f8c1d70: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 21:08:52.376806 10878 raft_consensus.cc:1261] T 403a964f77564be5a322d6b78d01318d P 1a3b362662f24aa5ba32e82b1e34ea5c [term 1 FOLLOWER]: Refusing update from remote peer 5bba75bcf25b4e7ca1c983368f8c1d70: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 21:08:52.377112 11083 consensus_queue.cc:1035] T 403a964f77564be5a322d6b78d01318d P 5bba75bcf25b4e7ca1c983368f8c1d70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "381653909834491e8889720c91a2985a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33987 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 21:08:52.377249 11079 consensus_queue.cc:1035] T 403a964f77564be5a322d6b78d01318d P 5bba75bcf25b4e7ca1c983368f8c1d70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1a3b362662f24aa5ba32e82b1e34ea5c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39367 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 21:08:52.379666 11087 mvcc.cc:205] Tried to move back new op lower bound from 7333003806212984832 to 7333003805914173440. Current Snapshot: MvccSnapshot[applied={T|T < 7333003806212984832}]
I20260924 21:08:52.379948 11088 mvcc.cc:205] Tried to move back new op lower bound from 7333003806212984832 to 7333003805914173440. Current Snapshot: MvccSnapshot[applied={T|T < 7333003806212984832}]
I20260924 21:08:52.381042 10598 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33692:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 21:08:52.381108 10598 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33692:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 21:08:52.381901 10598 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b8f951d2bbf64758834562fd3a76dee9: Sending DeleteTablet for 3 replicas of tablet 403a964f77564be5a322d6b78d01318d
I20260924 21:08:52.382218 10724 tablet_service.cc:1514] Processing DeleteTablet for tablet 403a964f77564be5a322d6b78d01318d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 21:08:52 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:39366
I20260924 21:08:52.382287 10858 tablet_service.cc:1514] Processing DeleteTablet for tablet 403a964f77564be5a322d6b78d01318d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 21:08:52 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:50470
I20260924 21:08:52.382373 10992 tablet_service.cc:1514] Processing DeleteTablet for tablet 403a964f77564be5a322d6b78d01318d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 21:08:52 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:42068
I20260924 21:08:52.382589 11097 tablet_replica.cc:330] T 403a964f77564be5a322d6b78d01318d P 1a3b362662f24aa5ba32e82b1e34ea5c: stopping tablet replica
I20260924 21:08:52.382682 11097 raft_consensus.cc:2229] T 403a964f77564be5a322d6b78d01318d P 1a3b362662f24aa5ba32e82b1e34ea5c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 21:08:52.382717 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAzNNjS/build/release/bin/kudu with pid 10661
I20260924 21:08:52.382795 11097 raft_consensus.cc:2258] T 403a964f77564be5a322d6b78d01318d P 1a3b362662f24aa5ba32e82b1e34ea5c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 21:08:52.382926 11098 tablet_replica.cc:330] T 403a964f77564be5a322d6b78d01318d P 381653909834491e8889720c91a2985a: stopping tablet replica
I20260924 21:08:52.382999 11098 raft_consensus.cc:2229] T 403a964f77564be5a322d6b78d01318d P 381653909834491e8889720c91a2985a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 21:08:52.383076 11098 raft_consensus.cc:2258] T 403a964f77564be5a322d6b78d01318d P 381653909834491e8889720c91a2985a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 21:08:52.383701 11097 ts_tablet_manager.cc:1905] T 403a964f77564be5a322d6b78d01318d P 1a3b362662f24aa5ba32e82b1e34ea5c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 21:08:52.384720 11098 ts_tablet_manager.cc:1905] T 403a964f77564be5a322d6b78d01318d P 381653909834491e8889720c91a2985a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 21:08:52.384953 11097 ts_tablet_manager.cc:1918] T 403a964f77564be5a322d6b78d01318d P 1a3b362662f24aa5ba32e82b1e34ea5c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 21:08:52.385025 11097 log.cc:1196] T 403a964f77564be5a322d6b78d01318d P 1a3b362662f24aa5ba32e82b1e34ea5c: Deleting WAL directory at /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790284076800039-392-0/minicluster-data/ts-1/wal/wals/403a964f77564be5a322d6b78d01318d
I20260924 21:08:52.385224 11097 ts_tablet_manager.cc:1939] T 403a964f77564be5a322d6b78d01318d P 1a3b362662f24aa5ba32e82b1e34ea5c: Deleting consensus metadata
I20260924 21:08:52.385681 10585 catalog_manager.cc:4841] TS 1a3b362662f24aa5ba32e82b1e34ea5c (127.0.98.2:39367): tablet 403a964f77564be5a322d6b78d01318d (table test-table [id=f19ef6c32df140d983c151a1cde690cb]) successfully deleted
I20260924 21:08:52.386453 11098 ts_tablet_manager.cc:1918] T 403a964f77564be5a322d6b78d01318d P 381653909834491e8889720c91a2985a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 21:08:52.386520 11098 log.cc:1196] T 403a964f77564be5a322d6b78d01318d P 381653909834491e8889720c91a2985a: Deleting WAL directory at /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790284076800039-392-0/minicluster-data/ts-2/wal/wals/403a964f77564be5a322d6b78d01318d
I20260924 21:08:52.386745 11098 ts_tablet_manager.cc:1939] T 403a964f77564be5a322d6b78d01318d P 381653909834491e8889720c91a2985a: Deleting consensus metadata
I20260924 21:08:52.387056 10585 catalog_manager.cc:4841] TS 381653909834491e8889720c91a2985a (127.0.98.3:33987): tablet 403a964f77564be5a322d6b78d01318d (table test-table [id=f19ef6c32df140d983c151a1cde690cb]) successfully deleted
W20260924 21:08:52.389022 10584 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:39831 (error 108)
I20260924 21:08:52.389277 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAzNNjS/build/release/bin/kudu with pid 10795
I20260924 21:08:52.389421 11066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:08:52.389300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39831 (local address 127.0.0.1:39380)
0924 21:08:52.389353 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0924 21:08:52.389390 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:39831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260924 21:08:52.390223 10584 catalog_manager.cc:4584] TS 5bba75bcf25b4e7ca1c983368f8c1d70 (127.0.98.1:39831): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 403a964f77564be5a322d6b78d01318d: Network error: Client connection negotiation failed: client connection to 127.0.98.1:39831: connect: Connection refused (error 111)
I20260924 21:08:52.393710 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAzNNjS/build/release/bin/kudu with pid 10929
I20260924 21:08:52.397778 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAzNNjS/build/release/bin/kudu with pid 10567
2026-09-24T21:08:52Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (1969 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790284076800039-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 21:08:52 dist-test-slave-c7tk krb5kdc[11102](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 21:08:52 dist-test-slave-c7tk krb5kdc[11102](info): set up 2 sockets
Sep 24 21:08:52 dist-test-slave-c7tk krb5kdc[11102](info): commencing operation
krb5kdc: starting...
W20260924 21:08:53.322309 392 mini_kdc.cc:121] Time spent starting KDC: real 0.905s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790284076800039-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790284076800039-392-0/krb5kdc/test-user.keytab.
Sep 24 21:08:53 dist-test-slave-c7tk krb5kdc[11102](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284133, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T21:08:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T21:08:53Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790284076800039-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790284076800039-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790284076800039-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790284076800039-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790284076800039-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790284076800039-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
I20260924 21:08:53.462440 392 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790284076800039-392-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskAzNNjS/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790284076800039-392-0/minicluster-data/postgres -l logfile start
2026-09-24 21:08:55.664 UTC [11132] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-24 21:08:55.664 UTC [11132] LOG: listening on IPv4 address "127.0.98.20", port 57153
2026-09-24 21:08:55.667 UTC [11132] LOG: listening on Unix socket "/tmp/.s.PGSQL.57153"
2026-09-24 21:08:55.698 UTC [11135] LOG: database system was shut down at 2026-09-24 21:08:54 UTC
2026-09-24 21:08:55.701 UTC [11132] LOG: database system is ready to accept connections
I20260924 21:08:56.541941 392 mini_postgres.cc:96] Postgres bound to 57153
2026-09-24 21:08:56.547 UTC [11143] FATAL: database "slave" does not exist
127.0.98.20:57153 - accepting connections
I20260924 21:08:56.547848 392 mini_ranger.cc:161] Starting Ranger...
I20260924 21:08:56.560319 392 mini_ranger.cc:84] Created miniranger Postgres user
I20260924 21:08:56.973080 392 mini_ranger.cc:87] Created ranger Postgres database
I20260924 21:08:56.973170 392 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790284076800039-392-0/minicluster-data/ranger-admin
2026-09-24 21:08:57,203 [I] DB FLAVOR :POSTGRES
2026-09-24 21:08:57,204 [I] --------- Verifying Ranger DB connection ---------
2026-09-24 21:08:57,204 [I] Checking connection..
2026-09-24 21:08:57,204 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAzNNjS/build/release/bin/postgresql.jar:/tmp/dist-test-taskAzNNjS/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:57153/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-24 21:08:57,436 [I] Checking connection passed.
2026-09-24 21:08:57,437 [I] --------- Verifying version history table ---------
2026-09-24 21:08:57,437 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAzNNjS/build/release/bin/postgresql.jar:/tmp/dist-test-taskAzNNjS/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:57153/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-24 21:08:57,660 [I] Table x_db_version_h does not exist in database ranger
2026-09-24 21:08:57,661 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAzNNjS/build/release/bin/postgresql.jar:/tmp/dist-test-taskAzNNjS/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:57153/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-24 21:08:57,857 [I] Table x_db_version_h does not exist in database ranger
2026-09-24 21:08:57,857 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-24 21:08:57,857 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAzNNjS/build/release/bin/postgresql.jar:/tmp/dist-test-taskAzNNjS/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:57153/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskAzNNjS/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-24 21:08:58.077 UTC [11207] WARNING: there is no transaction in progress
2026-09-24 21:08:58,083 [I] create_dbversion_catalog.sql file imported successfully
2026-09-24 21:08:58,083 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAzNNjS/build/release/bin/postgresql.jar:/tmp/dist-test-taskAzNNjS/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:57153/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-24 21:08:58,289 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-24 21:08:58,290 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-24 21:08:58,290 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAzNNjS/build/release/bin/postgresql.jar:/tmp/dist-test-taskAzNNjS/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:57153/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-24 21:08:58,486 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAzNNjS/build/release/bin/postgresql.jar:/tmp/dist-test-taskAzNNjS/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:57153/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-24 21:08:58,697 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAzNNjS/build/release/bin/postgresql.jar:/tmp/dist-test-taskAzNNjS/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:57153/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-c7tk.c.gcp-upstream.internal','N') ;"
2026-09-24 21:08:58,888 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAzNNjS/build/release/bin/postgresql.jar:/tmp/dist-test-taskAzNNjS/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:57153/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-24 21:08:59,082 [I] Table x_portal_user does not exist in database ranger
2026-09-24 21:08:59,082 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAzNNjS/build/release/bin/postgresql.jar:/tmp/dist-test-taskAzNNjS/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:57153/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-24 21:08:59,303 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-24 21:08:59,303 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAzNNjS/build/release/bin/postgresql.jar:/tmp/dist-test-taskAzNNjS/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:57153/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-24 21:08:59,533 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAzNNjS/build/release/bin/postgresql.jar:/tmp/dist-test-taskAzNNjS/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:57153/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-24 21:08:59,757 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-24 21:08:59,757 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAzNNjS/build/release/bin/postgresql.jar:/tmp/dist-test-taskAzNNjS/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:57153/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskAzNNjS/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-24 21:09:00.571 UTC [11315] WARNING: there is no transaction in progress
2026-09-24 21:09:00.585 UTC [11315] WARNING: there is no transaction in progress
2026-09-24 21:09:00.600 UTC [11315] WARNING: there is no transaction in progress
2026-09-24 21:09:00.614 UTC [11315] WARNING: there is no transaction in progress
2026-09-24 21:09:00.630 UTC [11315] WARNING: there is no transaction in progress
2026-09-24 21:09:00.645 UTC [11315] WARNING: there is no transaction in progress
2026-09-24 21:09:00.696 UTC [11315] WARNING: there is no transaction in progress
2026-09-24 21:09:00.701 UTC [11315] WARNING: there is no transaction in progress
2026-09-24 21:09:00.713 UTC [11315] WARNING: there is no transaction in progress
2026-09-24 21:09:00.722 UTC [11315] WARNING: there is no transaction in progress
2026-09-24 21:09:00.730 UTC [11315] WARNING: there is no transaction in progress
2026-09-24 21:09:00.742 UTC [11315] WARNING: there is no transaction in progress
2026-09-24 21:09:00.750 UTC [11315] WARNING: there is no transaction in progress
2026-09-24 21:09:00.755 UTC [11315] WARNING: there is no transaction in progress
2026-09-24 21:09:00.761 UTC [11315] WARNING: there is no transaction in progress
2026-09-24 21:09:00.766 UTC [11315] WARNING: there is no transaction in progress
2026-09-24 21:09:01,296 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-24 21:09:01,297 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAzNNjS/build/release/bin/postgresql.jar:/tmp/dist-test-taskAzNNjS/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:57153/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-24 21:09:01,514 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-24 21:09:01,514 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAzNNjS/build/release/bin/postgresql.jar:/tmp/dist-test-taskAzNNjS/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:57153/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-24 21:09:01,717 [I] Table x_portal_user already exists in database 'ranger'
2026-09-24 21:09:01,718 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAzNNjS/build/release/bin/postgresql.jar:/tmp/dist-test-taskAzNNjS/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:57153/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-24 21:09:01,941 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-24 21:09:01,942 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAzNNjS/build/release/bin/postgresql.jar:/tmp/dist-test-taskAzNNjS/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:57153/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-24 21:09:02,162 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAzNNjS/build/release/bin/postgresql.jar:/tmp/dist-test-taskAzNNjS/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:57153/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-24 21:09:02,373 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAzNNjS/build/release/bin/postgresql.jar:/tmp/dist-test-taskAzNNjS/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:57153/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-c7tk.c.gcp-upstream.internal';"
2026-09-24 21:09:02,569 [I] CORE_DB_SCHEMA import status has been updated
2026-09-24 21:09:02,570 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAzNNjS/build/release/bin/postgresql.jar:/tmp/dist-test-taskAzNNjS/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:57153/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-24 21:09:02,775 [I] DB_PATCHES have already been applied
I20260924 21:09:02.780730 392 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790284076800039-392-0/minicluster-data/ranger-admin:/tmp/dist-test-taskAzNNjS/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskAzNNjS/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskAzNNjS/build/release/bin/postgresql.jar:/tmp/dist-test-taskAzNNjS/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260924 21:09:02.780788 392 mini_ranger.cc:193] Using host: 127.0.98.20
I20260924 21:09:02.783494 392 mini_ranger.cc:227] Ranger admin URL: http://127.0.98.20:52937
Sep 24, 2026 9:09:03 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 24, 2026 9:09:03 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 24, 2026 9:09:03 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790284076800039-392-0/minicluster-data/ranger-admin/ews/webapp
Sep 24, 2026 9:09:03 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790284076800039-392-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 24, 2026 9:09:03 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790284076800039-392-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 24, 2026 9:09:03 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790284076800039-392-0/minicluster-data/ranger-admin/ews/webapp].
Sep 24, 2026 9:09:03 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260924 21:09:03.775868 392 mini_ranger.cc:160] Time spent starting Ranger: real 7.228s user 0.005s sys 0.002s
Sep 24, 2026 9:09:21 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 24, 2026 9:09:21 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.MetricsREST
Sep 24, 2026 9:09:21 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 24, 2026 9:09:21 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 24, 2026 9:09:21 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 24, 2026 9:09:21 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 24, 2026 9:09:21 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 24, 2026 9:09:21 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 24, 2026 9:09:21 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 24, 2026 9:09:21 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 24, 2026 9:09:21 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 24, 2026 9:09:21 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 24, 2026 9:09:21 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 24, 2026 9:09:21 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 24, 2026 9:09:21 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 24, 2026 9:09:21 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 24, 2026 9:09:21 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 24, 2026 9:09:22 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260924 21:09:22.954221 392 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790284076800039-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790284076800039-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.98.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790284076800039-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790284076800039-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 24 21:09:23 dist-test-slave-c7tk krb5kdc[11102](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284163, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260924 21:09:23.041761 392 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260924 21:09:23.053939 392 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260924 21:09:23.791548 392 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260924 21:09:23.791621 392 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790284076800039-392-0/minicluster-data/ranger-kms
I20260924 21:09:23.791633 392 mini_ranger_kms.cc:227] Using postgres at 127.0.98.20:57153
2026-09-24 21:09:23,948 [I] DB FLAVOR :POSTGRES
2026-09-24 21:09:23,948 [I] --------- Verifying Ranger DB connection ---------
2026-09-24 21:09:23,949 [I] Checking connection
2026-09-24 21:09:23,949 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAzNNjS/build/release/bin/postgresql.jar:/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790284076800039-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:57153/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-24 21:09:24,168 [I] connection success
2026-09-24 21:09:24,169 [I] --------- Verifying Ranger DB tables ---------
2026-09-24 21:09:24,169 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-24 21:09:24,169 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAzNNjS/build/release/bin/postgresql.jar:/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790284076800039-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:57153/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-24 21:09:24,368 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-24 21:09:24,368 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-24 21:09:24,369 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-24 21:09:24,369 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAzNNjS/build/release/bin/postgresql.jar:/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790284076800039-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:57153/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790284076800039-392-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-24 21:09:24,633 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260924 21:09:25.020205 392 mini_ranger_kms.cc:310] Created kms service
I20260924 21:09:25.200390 392 mini_ranger_kms.cc:326] Created kudu user
I20260924 21:09:25.236559 392 mini_ranger_kms.cc:343] Created rangerkms user
Sep 24 21:09:25 dist-test-slave-c7tk krb5kdc[11102](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284163, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260924 21:09:25.576197 392 mini_ranger_kms.cc:384] Added ranger policy
I20260924 21:09:25.576471 392 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790284076800039-392-0/minicluster-data/ranger-kms:/tmp/dist-test-taskAzNNjS/build/release/bin/postgresql.jar:/tmp/dist-test-taskAzNNjS/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskAzNNjS/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260924 21:09:25.576503 392 mini_ranger_kms.cc:240] Using host: 127.0.98.20
I20260924 21:09:25.579459 392 mini_ranger_kms.cc:276] Ranger KMS PID: 11495
I20260924 21:09:25.579572 392 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.98.20:54289
Sep 24, 2026 9:09:26 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 24, 2026 9:09:26 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 24, 2026 9:09:26 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790284076800039-392-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 24, 2026 9:09:26 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790284076800039-392-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 24, 2026 9:09:26 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790284076800039-392-0/minicluster-data/ranger-kms/ews/webapp].
Sep 24, 2026 9:09:26 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260924 21:09:27.535321 392 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 4.494s user 0.007s sys 0.136s
I20260924 21:09:27.535424 392 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260924 21:09:27.535501 392 mini_ranger_kms.cc:401] 127.0.98.20:54289/kms/v1/keys
Sep 24 21:09:27 dist-test-slave-c7tk krb5kdc[11102](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790284167, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 24 21:09:28 dist-test-slave-c7tk krb5kdc[11102](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284167, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 24 21:09:28 dist-test-slave-c7tk krb5kdc[11102](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284167, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 24 21:09:30 dist-test-slave-c7tk krb5kdc[11102](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284170, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790284076800039-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790284076800039-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 21:09:30.392647 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAzNNjS/build/release/bin/kudu
/tmp/dist-test-taskAzNNjS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790284076800039-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790284076800039-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790284076800039-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790284076800039-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46175
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790284076800039-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41073
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46175
--ranger_config_path=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790284076800039-392-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:54289/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790284076800039-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790284076800039-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790284076800039-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 21:09:30.496649 11546 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 21:09:30.496809 11546 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 21:09:30.496825 11546 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 21:09:30.497730 11546 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 21:09:30.497761 11546 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 21:09:30.497773 11546 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 21:09:30.497783 11546 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 21:09:30.497848 11546 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 21:09:30.499099 11546 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41073
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790284076800039-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790284076800039-392-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.98.20:54289/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46175
--ranger_config_path=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790284076800039-392-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790284076800039-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46175
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790284076800039-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11546
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790284076800039-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 19:49:06 UTC on ccycloud.quasar-vhqjoc.root.comops.site
build id 4670864
I20260924 21:09:30.499248 11546 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 21:09:30.499382 11546 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 21:09:30.503620 11551 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 21:09:30.503741 11546 server_base.cc:1034] running on GCE node
W20260924 21:09:30.503630 11552 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 21:09:30.503863 11554 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 21:09:30.504101 11546 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 21:09:30.504349 11546 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 21:09:30.505503 11546 hybrid_clock.cc:648] HybridClock initialized: now 1790284170505486 us; error 26 us; skew 500 ppm
Sep 24 21:09:30 dist-test-slave-c7tk krb5kdc[11102](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284170, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 21:09:30.511552 11546 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 21:09:30.511966 11546 webserver.cc:460] Webserver started at http://127.0.98.62:36091/ using document root <none> and password file <none>
I20260924 21:09:30.512172 11546 fs_manager.cc:359] Metadata directory not provided
I20260924 21:09:30.512233 11546 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 21:09:30.512403 11546 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 21:09:30 dist-test-slave-c7tk krb5kdc[11102](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284170, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260924 21:09:30.556318 11546 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790284076800039-392-0/minicluster-data/master-0/data/instance:
uuid: "0d641ffbd17441b59f7ba554c14dc6d0"
format_stamp: "Formatted at 2026-09-24 21:09:30 on dist-test-slave-c7tk"
server_key: "885b2f6b5b774290056692cd0c317f0a"
server_key_iv: "027d269fc205847557989594cc439e4f"
server_key_version: "kuduclusterkey@0"
I20260924 21:09:30.556707 11546 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790284076800039-392-0/minicluster-data/master-0/wal/instance:
uuid: "0d641ffbd17441b59f7ba554c14dc6d0"
format_stamp: "Formatted at 2026-09-24 21:09:30 on dist-test-slave-c7tk"
server_key: "885b2f6b5b774290056692cd0c317f0a"
server_key_iv: "027d269fc205847557989594cc439e4f"
server_key_version: "kuduclusterkey@0"
I20260924 21:09:30.558964 11546 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.003s
I20260924 21:09:30.576061 11561 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 21:09:30.576278 11546 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 21:09:30.576357 11546 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790284076800039-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790284076800039-392-0/minicluster-data/master-0/wal
uuid: "0d641ffbd17441b59f7ba554c14dc6d0"
format_stamp: "Formatted at 2026-09-24 21:09:30 on dist-test-slave-c7tk"
server_key: "885b2f6b5b774290056692cd0c317f0a"
server_key_iv: "027d269fc205847557989594cc439e4f"
server_key_version: "kuduclusterkey@0"
I20260924 21:09:30.576449 11546 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790284076800039-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790284076800039-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790284076800039-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 21:09:30.595108 11546 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 21:09:30.596915 11546 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 21:09:30.597007 11546 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 21:09:30.600798 11546 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:46175
I20260924 21:09:30.600863 11613 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:46175 every 8 connection(s)
I20260924 21:09:30.601141 11546 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790284076800039-392-0/minicluster-data/master-0/data/info.pb
I20260924 21:09:30.601686 11614 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 21:09:30.603668 11614 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0d641ffbd17441b59f7ba554c14dc6d0: Bootstrap starting.
I20260924 21:09:30.604205 11614 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0d641ffbd17441b59f7ba554c14dc6d0: Neither blocks nor log segments found. Creating new log.
I20260924 21:09:30.604481 11614 log.cc:824] T 00000000000000000000000000000000 P 0d641ffbd17441b59f7ba554c14dc6d0: Log is configured to *not* fsync() on all Append() calls
I20260924 21:09:30.605083 11614 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0d641ffbd17441b59f7ba554c14dc6d0: No bootstrap required, opened a new log
I20260924 21:09:30.607460 11614 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0d641ffbd17441b59f7ba554c14dc6d0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d641ffbd17441b59f7ba554c14dc6d0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46175 } }
I20260924 21:09:30.607584 11614 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0d641ffbd17441b59f7ba554c14dc6d0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 21:09:30.607604 11614 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0d641ffbd17441b59f7ba554c14dc6d0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d641ffbd17441b59f7ba554c14dc6d0, State: Initialized, Role: FOLLOWER
I20260924 21:09:30.607752 11614 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0d641ffbd17441b59f7ba554c14dc6d0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d641ffbd17441b59f7ba554c14dc6d0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46175 } }
I20260924 21:09:30.607823 11614 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0d641ffbd17441b59f7ba554c14dc6d0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 21:09:30.607861 11614 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0d641ffbd17441b59f7ba554c14dc6d0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 21:09:30.607909 11614 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0d641ffbd17441b59f7ba554c14dc6d0 [term 0 FOLLOWER]: Advancing to term 1
I20260924 21:09:30.607970 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAzNNjS/build/release/bin/kudu as pid 11546
I20260924 21:09:30.608062 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790284076800039-392-0/minicluster-data/master-0/wal/instance
I20260924 21:09:30.608237 392 external_mini_cluster.cc:1428] Setting key a2710541715d68ba2f4cb8e7261b5520
I20260924 21:09:30.608505 11614 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0d641ffbd17441b59f7ba554c14dc6d0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d641ffbd17441b59f7ba554c14dc6d0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46175 } }
I20260924 21:09:30.608621 11614 leader_election.cc:302] T 00000000000000000000000000000000 P 0d641ffbd17441b59f7ba554c14dc6d0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0d641ffbd17441b59f7ba554c14dc6d0; no voters:
I20260924 21:09:30.608793 11614 leader_election.cc:288] T 00000000000000000000000000000000 P 0d641ffbd17441b59f7ba554c14dc6d0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 21:09:30.608822 11618 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0d641ffbd17441b59f7ba554c14dc6d0 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 21:09:30.608958 11618 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0d641ffbd17441b59f7ba554c14dc6d0 [term 1 LEADER]: Becoming Leader. State: Replica: 0d641ffbd17441b59f7ba554c14dc6d0, State: Running, Role: LEADER
I20260924 21:09:30.608994 11614 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0d641ffbd17441b59f7ba554c14dc6d0 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 21:09:30.609090 11618 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0d641ffbd17441b59f7ba554c14dc6d0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d641ffbd17441b59f7ba554c14dc6d0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46175 } }
I20260924 21:09:30.610081 11620 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d641ffbd17441b59f7ba554c14dc6d0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0d641ffbd17441b59f7ba554c14dc6d0. Latest consensus state: current_term: 1 leader_uuid: "0d641ffbd17441b59f7ba554c14dc6d0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d641ffbd17441b59f7ba554c14dc6d0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46175 } } }
I20260924 21:09:30.610145 11620 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d641ffbd17441b59f7ba554c14dc6d0 [sys.catalog]: This master's current role is: LEADER
I20260924 21:09:30.610493 11614 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790284076800039-392-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260924 21:09:30.610570 11621 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d641ffbd17441b59f7ba554c14dc6d0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0d641ffbd17441b59f7ba554c14dc6d0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d641ffbd17441b59f7ba554c14dc6d0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46175 } } }
I20260924 21:09:30.610636 11621 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d641ffbd17441b59f7ba554c14dc6d0 [sys.catalog]: This master's current role is: LEADER
I20260924 21:09:30.610733 11631 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 21:09:30.612932 11631 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 21:09:30.615012 11631 catalog_manager.cc:1347] Generated new cluster ID: db46b0e3d10541cb95057a3e181439c4
I20260924 21:09:30.615059 11631 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 24 21:09:30 dist-test-slave-c7tk krb5kdc[11102](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284170, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 21:09:30.620245 11619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:09:30.608704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58614 (local address 127.0.98.62:46175)
0924 21:09:30.608867 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0924 21:09:30.608876 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 21:09:30.608904 (+ 28us) server_negotiation.cc:407] Connection header received
0924 21:09:30.608998 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:09:30.609004 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:09:30.609074 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:09:30.609205 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:09:30.609952 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:09:30.612569 (+ 2617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:09:30.613408 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:09:30.614092 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:09:30.614206 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:09:30.616657 (+ 2451us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 21:09:30.616661 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 21:09:30.616661 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 21:09:30.616671 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0924 21:09:30.618408 (+ 1737us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:09:30.618759 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:09:30.618761 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:09:30.618761 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:09:30.618843 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:09:30.619023 (+ 180us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:09:30.619023 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:09:30.619024 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:09:30.619297 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 21:09:30.619872 (+ 575us) server_negotiation.cc:1035] Waiting for connection context
0924 21:09:30.619935 (+ 63us) server_negotiation.cc:299] Negotiation successful
0924 21:09:30.620043 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":67,"threads_started":1}
I20260924 21:09:30.636037 11631 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 21:09:30.636711 11631 catalog_manager.cc:1504] Loading token signing keys...
I20260924 21:09:30.643407 11631 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0d641ffbd17441b59f7ba554c14dc6d0: Generated new TSK 0
I20260924 21:09:30.643568 11631 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790284076800039-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790284076800039-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260924 21:09:31.224061 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAzNNjS/build/release/bin/kudu
/tmp/dist-test-taskAzNNjS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790284076800039-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790284076800039-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790284076800039-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790284076800039-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:54289/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46175
--keytab_file=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790284076800039-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41073
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790284076800039-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790284076800039-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790284076800039-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 21:09:31.324100 11659 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 21:09:31.324276 11659 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 21:09:31.324295 11659 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 21:09:31.325345 11659 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 21:09:31.325388 11659 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 21:09:31.325413 11659 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 21:09:31.326879 11659 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41073
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790284076800039-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790284076800039-392-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.98.20:54289/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790284076800039-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790284076800039-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46175
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11659
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790284076800039-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 19:49:06 UTC on ccycloud.quasar-vhqjoc.root.comops.site
build id 4670864
I20260924 21:09:31.327121 11659 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 21:09:31.327358 11659 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 21:09:31.332729 11667 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 21:09:31.332707 11665 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 21:09:31.332785 11659 server_base.cc:1034] running on GCE node
W20260924 21:09:31.332877 11664 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 21:09:31.333078 11659 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 21:09:31.333328 11659 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 21:09:31.334482 11659 hybrid_clock.cc:648] HybridClock initialized: now 1790284171334472 us; error 34 us; skew 500 ppm
Sep 24 21:09:31 dist-test-slave-c7tk krb5kdc[11102](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284171, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 21:09:31.336910 11659 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 21:09:31.337281 11659 webserver.cc:460] Webserver started at http://127.0.98.1:36933/ using document root <none> and password file <none>
I20260924 21:09:31.337489 11659 fs_manager.cc:359] Metadata directory not provided
I20260924 21:09:31.337551 11659 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 21:09:31.337666 11659 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 21:09:31 dist-test-slave-c7tk krb5kdc[11102](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284171, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260924 21:09:31.360402 11659 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790284076800039-392-0/minicluster-data/ts-0/data/instance:
uuid: "07caf695041b44299e67de3255b71eda"
format_stamp: "Formatted at 2026-09-24 21:09:31 on dist-test-slave-c7tk"
server_key: "e2fa46cd544289e0bba98be05d550015"
server_key_iv: "2d903764a3a4f25f4afe048158537833"
server_key_version: "kuduclusterkey@0"
I20260924 21:09:31.360821 11659 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790284076800039-392-0/minicluster-data/ts-0/wal/instance:
uuid: "07caf695041b44299e67de3255b71eda"
format_stamp: "Formatted at 2026-09-24 21:09:31 on dist-test-slave-c7tk"
server_key: "e2fa46cd544289e0bba98be05d550015"
server_key_iv: "2d903764a3a4f25f4afe048158537833"
server_key_version: "kuduclusterkey@0"
I20260924 21:09:31.362473 11659 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 21:09:31.375934 11676 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 21:09:31.376132 11659 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260924 21:09:31.376226 11659 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790284076800039-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790284076800039-392-0/minicluster-data/ts-0/wal
uuid: "07caf695041b44299e67de3255b71eda"
format_stamp: "Formatted at 2026-09-24 21:09:31 on dist-test-slave-c7tk"
server_key: "e2fa46cd544289e0bba98be05d550015"
server_key_iv: "2d903764a3a4f25f4afe048158537833"
server_key_version: "kuduclusterkey@0"
I20260924 21:09:31.376340 11659 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790284076800039-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790284076800039-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790284076800039-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 21:09:31.394904 11659 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 21:09:31.397892 11659 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 21:09:31.398065 11659 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 21:09:31.398300 11659 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 21:09:31.398625 11659 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 21:09:31.398684 11659 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 21:09:31.398746 11659 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 21:09:31.398788 11659 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 21:09:31.406679 11659 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:36413
I20260924 21:09:31.406756 11789 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:36413 every 8 connection(s)
I20260924 21:09:31.407101 11659 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790284076800039-392-0/minicluster-data/ts-0/data/info.pb
I20260924 21:09:31.409416 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAzNNjS/build/release/bin/kudu as pid 11659
I20260924 21:09:31.409554 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790284076800039-392-0/minicluster-data/ts-0/wal/instance
I20260924 21:09:31.409823 392 external_mini_cluster.cc:1428] Setting key c8d06ce77e68a3ca9183a1ca777f2a3f
Sep 24 21:09:31 dist-test-slave-c7tk krb5kdc[11102](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284171, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260924 21:09:31.419993 11793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:09:31.407842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:55943 (local address 127.0.98.62:46175)
0924 21:09:31.408031 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0924 21:09:31.408033 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 21:09:31.408082 (+ 49us) server_negotiation.cc:407] Connection header received
0924 21:09:31.408730 (+ 648us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:09:31.408731 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:09:31.408743 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:09:31.408806 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:09:31.409457 (+ 651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:09:31.410234 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:09:31.413026 (+ 2792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:09:31.413365 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:09:31.413440 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:09:31.416227 (+ 2787us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 21:09:31.416229 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 21:09:31.416229 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 21:09:31.416238 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 21:09:31.418105 (+ 1867us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:09:31.418964 (+ 859us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:09:31.418965 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:09:31.418966 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:09:31.419040 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:09:31.419304 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:09:31.419305 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:09:31.419306 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:09:31.419438 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 21:09:31.419499 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0924 21:09:31.419830 (+ 331us) server_negotiation.cc:299] Negotiation successful
0924 21:09:31.419906 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":154,"thread_start_us":70,"threads_started":1}
I20260924 21:09:31.420146 11792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:09:31.407773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46175 (local address 127.0.98.1:55943)
0924 21:09:31.407967 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0924 21:09:31.408009 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 21:09:31.408633 (+ 624us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 21:09:31.408880 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 21:09:31.408884 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 21:09:31.409098 (+ 214us) client_negotiation.cc:189] Negotiated authn=SASL
0924 21:09:31.409348 (+ 250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:09:31.409354 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:09:31.410355 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:09:31.410356 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:09:31.412933 (+ 2577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:09:31.412935 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:09:31.413462 (+ 527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:09:31.413463 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:09:31.413620 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:09:31.414326 (+ 706us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 21:09:31.414336 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 21:09:31.416110 (+ 1774us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 21:09:31.418205 (+ 2095us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 21:09:31.418212 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 21:09:31.418214 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 21:09:31.418875 (+ 661us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 21:09:31.419104 (+ 229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 21:09:31.419104 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 21:09:31.419105 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 21:09:31.419225 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 21:09:31.419541 (+ 316us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 21:09:31.419548 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 21:09:31.419693 (+ 145us) client_negotiation.cc:769] Sending connection context
0924 21:09:31.419807 (+ 114us) client_negotiation.cc:240] Negotiation successful
0924 21:09:31.419937 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":83,"threads_started":1}
I20260924 21:09:31.420816 11790 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46175
I20260924 21:09:31.420921 11790 heartbeater.cc:464] Registering TS with master...
I20260924 21:09:31.421099 11790 heartbeater.cc:511] Master 127.0.98.62:46175 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260924 21:09:31.421579 11578 ts_manager.cc:194] Registered new tserver with Master: 07caf695041b44299e67de3255b71eda (127.0.98.1:36413)
I20260924 21:09:31.422425 11578 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:55943
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790284076800039-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790284076800039-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260924 21:09:31.436754 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAzNNjS/build/release/bin/kudu
/tmp/dist-test-taskAzNNjS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790284076800039-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790284076800039-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790284076800039-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790284076800039-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:54289/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46175
--keytab_file=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790284076800039-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41073
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790284076800039-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790284076800039-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790284076800039-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 21:09:31.529914 11796 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 21:09:31.530119 11796 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 21:09:31.530158 11796 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 21:09:31.531209 11796 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 21:09:31.531268 11796 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 21:09:31.531306 11796 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 21:09:31.532704 11796 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41073
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790284076800039-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790284076800039-392-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.98.20:54289/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790284076800039-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790284076800039-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46175
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11796
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790284076800039-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 19:49:06 UTC on ccycloud.quasar-vhqjoc.root.comops.site
build id 4670864
I20260924 21:09:31.532917 11796 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
Sep 24 21:09:31 dist-test-slave-c7tk krb5kdc[11102](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790284171, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 21:09:31.533628 11796 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 21:09:31.539423 11804 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 21:09:31.539976 11806 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 21:09:31.540179 11803 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 21:09:31.540747 11796 server_base.cc:1034] running on GCE node
I20260924 21:09:31.540932 11796 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 21:09:31.541167 11796 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 21:09:31.542339 11796 hybrid_clock.cc:648] HybridClock initialized: now 1790284171542315 us; error 31 us; skew 500 ppm
Sep 24 21:09:31 dist-test-slave-c7tk krb5kdc[11102](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284171, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 21:09:31.545224 11796 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 21:09:31.545761 11796 webserver.cc:460] Webserver started at http://127.0.98.2:44219/ using document root <none> and password file <none>
I20260924 21:09:31.546028 11796 fs_manager.cc:359] Metadata directory not provided
I20260924 21:09:31.546089 11796 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 21:09:31.546211 11796 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 21:09:31 dist-test-slave-c7tk krb5kdc[11102](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284171, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260924 21:09:31.574203 11796 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790284076800039-392-0/minicluster-data/ts-1/data/instance:
uuid: "1205036197a943938239309e0f2f4ce9"
format_stamp: "Formatted at 2026-09-24 21:09:31 on dist-test-slave-c7tk"
server_key: "eedea1a6cf2b8c73bbca942188ce8345"
server_key_iv: "7469f24d66f6d9d49716389feaa269b2"
server_key_version: "kuduclusterkey@0"
I20260924 21:09:31.574589 11796 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790284076800039-392-0/minicluster-data/ts-1/wal/instance:
uuid: "1205036197a943938239309e0f2f4ce9"
format_stamp: "Formatted at 2026-09-24 21:09:31 on dist-test-slave-c7tk"
server_key: "eedea1a6cf2b8c73bbca942188ce8345"
server_key_iv: "7469f24d66f6d9d49716389feaa269b2"
server_key_version: "kuduclusterkey@0"
I20260924 21:09:31.576112 11796 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 21:09:31.586980 11813 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 21:09:31.587134 11796 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260924 21:09:31.587217 11796 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790284076800039-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790284076800039-392-0/minicluster-data/ts-1/wal
uuid: "1205036197a943938239309e0f2f4ce9"
format_stamp: "Formatted at 2026-09-24 21:09:31 on dist-test-slave-c7tk"
server_key: "eedea1a6cf2b8c73bbca942188ce8345"
server_key_iv: "7469f24d66f6d9d49716389feaa269b2"
server_key_version: "kuduclusterkey@0"
I20260924 21:09:31.587314 11796 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790284076800039-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790284076800039-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790284076800039-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 21:09:31.600291 11796 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 21:09:31.604839 11796 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 21:09:31.604980 11796 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 21:09:31.605365 11796 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 21:09:31.605729 11796 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 21:09:31.605773 11796 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 21:09:31.605823 11796 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 21:09:31.605871 11796 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 21:09:31.612203 11796 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:40363
I20260924 21:09:31.612584 11796 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790284076800039-392-0/minicluster-data/ts-1/data/info.pb
I20260924 21:09:31.615442 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAzNNjS/build/release/bin/kudu as pid 11796
I20260924 21:09:31.615540 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790284076800039-392-0/minicluster-data/ts-1/wal/instance
I20260924 21:09:31.615743 392 external_mini_cluster.cc:1428] Setting key c4f48b8ce501a65991e0be0ba2e4a96f
I20260924 21:09:31.617995 11926 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:40363 every 8 connection(s)
Sep 24 21:09:31 dist-test-slave-c7tk krb5kdc[11102](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284171, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 21:09:31.629844 11793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:09:31.618756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:48991 (local address 127.0.98.62:46175)
0924 21:09:31.618798 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0924 21:09:31.618800 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 21:09:31.621466 (+ 2666us) server_negotiation.cc:407] Connection header received
0924 21:09:31.622005 (+ 539us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:09:31.622007 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:09:31.622019 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:09:31.622048 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:09:31.622800 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:09:31.623259 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:09:31.624129 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:09:31.624401 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:09:31.624460 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:09:31.626859 (+ 2399us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 21:09:31.626860 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 21:09:31.626860 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 21:09:31.626866 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 21:09:31.628435 (+ 1569us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:09:31.628893 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:09:31.628894 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:09:31.628895 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:09:31.628948 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:09:31.629216 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:09:31.629217 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:09:31.629217 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:09:31.629326 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 21:09:31.629361 (+ 35us) server_negotiation.cc:1035] Waiting for connection context
0924 21:09:31.629672 (+ 311us) server_negotiation.cc:299] Negotiation successful
0924 21:09:31.629747 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260924 21:09:31.630264 11930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:09:31.618531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46175 (local address 127.0.98.2:48991)
0924 21:09:31.621343 (+ 2812us) negotiation.cc:107] Waiting for socket to connect
0924 21:09:31.621376 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0924 21:09:31.621883 (+ 507us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 21:09:31.622212 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 21:09:31.622216 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 21:09:31.622438 (+ 222us) client_negotiation.cc:189] Negotiated authn=SASL
0924 21:09:31.622694 (+ 256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:09:31.622701 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:09:31.623340 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:09:31.623340 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:09:31.624043 (+ 703us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:09:31.624045 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:09:31.624520 (+ 475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:09:31.624520 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:09:31.624674 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:09:31.625200 (+ 526us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 21:09:31.625210 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 21:09:31.626767 (+ 1557us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 21:09:31.628522 (+ 1755us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 21:09:31.628527 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 21:09:31.628529 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 21:09:31.628807 (+ 278us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 21:09:31.629013 (+ 206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 21:09:31.629013 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 21:09:31.629014 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 21:09:31.629152 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 21:09:31.629398 (+ 246us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 21:09:31.629403 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 21:09:31.629545 (+ 142us) client_negotiation.cc:769] Sending connection context
0924 21:09:31.629933 (+ 388us) client_negotiation.cc:240] Negotiation successful
0924 21:09:31.630074 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2775,"thread_start_us":67,"threads_started":1}
I20260924 21:09:31.630776 11927 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46175
I20260924 21:09:31.630889 11927 heartbeater.cc:464] Registering TS with master...
I20260924 21:09:31.631060 11927 heartbeater.cc:511] Master 127.0.98.62:46175 requested a full tablet report, sending...
I20260924 21:09:31.631392 11578 ts_manager.cc:194] Registered new tserver with Master: 1205036197a943938239309e0f2f4ce9 (127.0.98.2:40363)
I20260924 21:09:31.631812 11578 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:48991
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790284076800039-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790284076800039-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260924 21:09:31.657437 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAzNNjS/build/release/bin/kudu
/tmp/dist-test-taskAzNNjS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790284076800039-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790284076800039-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790284076800039-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790284076800039-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:54289/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46175
--keytab_file=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790284076800039-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41073
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790284076800039-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790284076800039-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790284076800039-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 21:09:31.762836 11933 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 21:09:31.763085 11933 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 21:09:31.763123 11933 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 21:09:31.764626 11933 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 21:09:31.764681 11933 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 21:09:31.764715 11933 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 21:09:31.766749 11933 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41073
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790284076800039-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790284076800039-392-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.98.20:54289/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790284076800039-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790284076800039-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46175
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11933
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790284076800039-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 19:49:06 UTC on ccycloud.quasar-vhqjoc.root.comops.site
build id 4670864
I20260924 21:09:31.766954 11933 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 21:09:31.767282 11933 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 21:09:31.774186 11941 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 21:09:31.774335 11938 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 21:09:31.774299 11939 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 21:09:31.774796 11933 server_base.cc:1034] running on GCE node
I20260924 21:09:31.774973 11933 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 21:09:31.775198 11933 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 21:09:31.776366 11933 hybrid_clock.cc:648] HybridClock initialized: now 1790284171776344 us; error 33 us; skew 500 ppm
Sep 24 21:09:31 dist-test-slave-c7tk krb5kdc[11102](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284171, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 21:09:31.779700 11933 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 21:09:31.780136 11933 webserver.cc:460] Webserver started at http://127.0.98.3:43851/ using document root <none> and password file <none>
I20260924 21:09:31.780349 11933 fs_manager.cc:359] Metadata directory not provided
I20260924 21:09:31.780395 11933 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 21:09:31.780501 11933 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 21:09:31 dist-test-slave-c7tk krb5kdc[11102](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284171, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260924 21:09:31.795696 11933 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790284076800039-392-0/minicluster-data/ts-2/data/instance:
uuid: "f5b5f663ef0443518d8025f0ef3e75ae"
format_stamp: "Formatted at 2026-09-24 21:09:31 on dist-test-slave-c7tk"
server_key: "c97dd785792b4486f5835f08182c9c5d"
server_key_iv: "64e7b214b69133812ae7cc95116830d2"
server_key_version: "kuduclusterkey@0"
I20260924 21:09:31.796204 11933 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790284076800039-392-0/minicluster-data/ts-2/wal/instance:
uuid: "f5b5f663ef0443518d8025f0ef3e75ae"
format_stamp: "Formatted at 2026-09-24 21:09:31 on dist-test-slave-c7tk"
server_key: "c97dd785792b4486f5835f08182c9c5d"
server_key_iv: "64e7b214b69133812ae7cc95116830d2"
server_key_version: "kuduclusterkey@0"
I20260924 21:09:31.797827 11933 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 21:09:31.821996 11948 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 21:09:31.822256 11933 fs_manager.cc:725] Time spent opening block manager: real 0.008s user 0.000s sys 0.001s
I20260924 21:09:31.822366 11933 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790284076800039-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790284076800039-392-0/minicluster-data/ts-2/wal
uuid: "f5b5f663ef0443518d8025f0ef3e75ae"
format_stamp: "Formatted at 2026-09-24 21:09:31 on dist-test-slave-c7tk"
server_key: "c97dd785792b4486f5835f08182c9c5d"
server_key_iv: "64e7b214b69133812ae7cc95116830d2"
server_key_version: "kuduclusterkey@0"
I20260924 21:09:31.822465 11933 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790284076800039-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790284076800039-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790284076800039-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 21:09:31.847924 11933 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 21:09:31.850196 11933 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 21:09:31.850327 11933 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 21:09:31.850500 11933 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 21:09:31.850876 11933 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 21:09:31.850940 11933 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 21:09:31.850967 11933 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 21:09:31.850984 11933 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 21:09:31.858207 11933 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:35101
I20260924 21:09:31.858297 12061 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:35101 every 8 connection(s)
I20260924 21:09:31.858618 11933 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790284076800039-392-0/minicluster-data/ts-2/data/info.pb
I20260924 21:09:31.866011 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAzNNjS/build/release/bin/kudu as pid 11933
I20260924 21:09:31.866106 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790284076800039-392-0/minicluster-data/ts-2/wal/instance
I20260924 21:09:31.866297 392 external_mini_cluster.cc:1428] Setting key e357fdaf53016eacdfa975223206b677
Sep 24 21:09:31 dist-test-slave-c7tk krb5kdc[11102](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284171, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 21:09:31.900102 11793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:09:31.877741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:43251 (local address 127.0.98.62:46175)
0924 21:09:31.877780 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0924 21:09:31.877782 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 21:09:31.878074 (+ 292us) server_negotiation.cc:407] Connection header received
0924 21:09:31.878719 (+ 645us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:09:31.878720 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:09:31.878731 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:09:31.878758 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:09:31.879505 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:09:31.880241 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:09:31.890384 (+ 10143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:09:31.890691 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:09:31.890736 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:09:31.894551 (+ 3815us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 21:09:31.894553 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 21:09:31.894553 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 21:09:31.894560 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 21:09:31.896282 (+ 1722us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:09:31.899154 (+ 2872us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:09:31.899156 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:09:31.899156 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:09:31.899213 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:09:31.899454 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:09:31.899455 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:09:31.899455 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:09:31.899564 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 21:09:31.899614 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0924 21:09:31.899932 (+ 318us) server_negotiation.cc:299] Negotiation successful
0924 21:09:31.900005 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 21:09:31.900861 12064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:09:31.877511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46175 (local address 127.0.98.3:43251)
0924 21:09:31.877955 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0924 21:09:31.877995 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 21:09:31.878610 (+ 615us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 21:09:31.878930 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 21:09:31.878934 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 21:09:31.879159 (+ 225us) client_negotiation.cc:189] Negotiated authn=SASL
0924 21:09:31.879410 (+ 251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:09:31.879416 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:09:31.883833 (+ 4417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:09:31.883834 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:09:31.890286 (+ 6452us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:09:31.890289 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:09:31.890827 (+ 538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:09:31.890828 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:09:31.891480 (+ 652us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:09:31.892575 (+ 1095us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 21:09:31.892584 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 21:09:31.894421 (+ 1837us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 21:09:31.896814 (+ 2393us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 21:09:31.896820 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 21:09:31.896822 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 21:09:31.897125 (+ 303us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 21:09:31.899277 (+ 2152us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 21:09:31.899278 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 21:09:31.899278 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 21:09:31.899392 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 21:09:31.899643 (+ 251us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 21:09:31.899648 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 21:09:31.899798 (+ 150us) client_negotiation.cc:769] Sending connection context
0924 21:09:31.900469 (+ 671us) client_negotiation.cc:240] Negotiation successful
0924 21:09:31.900625 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":410,"thread_start_us":67,"threads_started":1}
I20260924 21:09:31.902328 12062 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46175
I20260924 21:09:31.902513 12062 heartbeater.cc:464] Registering TS with master...
I20260924 21:09:31.902810 12062 heartbeater.cc:511] Master 127.0.98.62:46175 requested a full tablet report, sending...
I20260924 21:09:31.903294 11578 ts_manager.cc:194] Registered new tserver with Master: f5b5f663ef0443518d8025f0ef3e75ae (127.0.98.3:35101)
I20260924 21:09:31.903716 11578 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:43251
I20260924 21:09:31.912572 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 21:09:31.927145 11793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:09:31.914958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58616 (local address 127.0.98.62:46175)
0924 21:09:31.914991 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0924 21:09:31.914993 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 21:09:31.916248 (+ 1255us) server_negotiation.cc:407] Connection header received
0924 21:09:31.916347 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:09:31.916348 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:09:31.916358 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:09:31.916387 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:09:31.917287 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:09:31.917953 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:09:31.919060 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:09:31.919296 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:09:31.919335 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:09:31.920914 (+ 1579us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 21:09:31.920915 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 21:09:31.920915 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 21:09:31.920921 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 21:09:31.922437 (+ 1516us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:09:31.924630 (+ 2193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:09:31.924632 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:09:31.924632 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:09:31.924686 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:09:31.925802 (+ 1116us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:09:31.925804 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:09:31.925804 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:09:31.925980 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 21:09:31.926088 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0924 21:09:31.926915 (+ 827us) server_negotiation.cc:299] Negotiation successful
0924 21:09:31.927074 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260924 21:09:31.928977 11578 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58616:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 21:09:31.931041 11578 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 21:09:31.946774 12077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:09:31.941169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36413 (local address 127.0.0.1:50446)
0924 21:09:31.942236 (+ 1067us) negotiation.cc:107] Waiting for socket to connect
0924 21:09:31.942242 (+ 6us) client_negotiation.cc:174] Beginning negotiation
0924 21:09:31.942272 (+ 30us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 21:09:31.942525 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 21:09:31.942528 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 21:09:31.942529 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 21:09:31.942588 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:09:31.942589 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:09:31.943926 (+ 1337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:09:31.943927 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:09:31.945104 (+ 1177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:09:31.945106 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:09:31.946195 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:09:31.946196 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:09:31.946284 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:09:31.946297 (+ 13us) client_negotiation.cc:769] Sending connection context
0924 21:09:31.946653 (+ 356us) client_negotiation.cc:240] Negotiation successful
0924 21:09:31.946708 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1058,"thread_start_us":43,"threads_started":1}
I20260924 21:09:31.947163 12080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:09:31.941325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50446 (local address 127.0.98.1:36413)
0924 21:09:31.942103 (+ 778us) server_negotiation.cc:206] Beginning negotiation
0924 21:09:31.942108 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 21:09:31.942312 (+ 204us) server_negotiation.cc:407] Connection header received
0924 21:09:31.942361 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:09:31.942363 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:09:31.942377 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:09:31.942448 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 21:09:31.942656 (+ 208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:09:31.943856 (+ 1200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:09:31.945223 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:09:31.946039 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:09:31.946107 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:09:31.946162 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0924 21:09:31.947045 (+ 883us) server_negotiation.cc:299] Negotiation successful
0924 21:09:31.947085 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":746,"thread_start_us":41,"threads_started":1}
I20260924 21:09:31.947966 11724 tablet_service.cc:1467] Processing CreateTablet for tablet 704b37e2f1bb4a458ac07a91acc760f5 (DEFAULT_TABLE table=test-table [id=a764df1a7fe047a0ac4823d4d698a5d7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 21:09:31.948303 11724 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 704b37e2f1bb4a458ac07a91acc760f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 21:09:31.953007 12081 tablet_bootstrap.cc:492] T 704b37e2f1bb4a458ac07a91acc760f5 P 07caf695041b44299e67de3255b71eda: Bootstrap starting.
I20260924 21:09:31.953781 12081 tablet_bootstrap.cc:654] T 704b37e2f1bb4a458ac07a91acc760f5 P 07caf695041b44299e67de3255b71eda: Neither blocks nor log segments found. Creating new log.
I20260924 21:09:31.956130 12081 log.cc:824] T 704b37e2f1bb4a458ac07a91acc760f5 P 07caf695041b44299e67de3255b71eda: Log is configured to *not* fsync() on all Append() calls
I20260924 21:09:31.957077 12075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:09:31.941054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40363 (local address 127.0.0.1:36278)
0924 21:09:31.942144 (+ 1090us) negotiation.cc:107] Waiting for socket to connect
0924 21:09:31.942151 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0924 21:09:31.942183 (+ 32us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 21:09:31.951769 (+ 9586us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 21:09:31.951770 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 21:09:31.951772 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 21:09:31.951821 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:09:31.951822 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:09:31.953943 (+ 2121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:09:31.953944 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:09:31.955065 (+ 1121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:09:31.955068 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:09:31.956877 (+ 1809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:09:31.956878 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:09:31.956941 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:09:31.956946 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 21:09:31.956970 (+ 24us) client_negotiation.cc:240] Negotiation successful
0924 21:09:31.957001 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1079,"thread_start_us":51,"threads_started":1}
I20260924 21:09:31.957526 12078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:09:31.940836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:35101 (local address 127.0.0.1:41004)
0924 21:09:31.941874 (+ 1038us) negotiation.cc:107] Waiting for socket to connect
0924 21:09:31.941918 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0924 21:09:31.941976 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 21:09:31.954060 (+ 12084us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 21:09:31.954061 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 21:09:31.954063 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 21:09:31.954206 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:09:31.954209 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:09:31.955328 (+ 1119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:09:31.955329 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:09:31.956450 (+ 1121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:09:31.956453 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:09:31.957332 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:09:31.957333 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:09:31.957394 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:09:31.957397 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 21:09:31.957436 (+ 39us) client_negotiation.cc:240] Negotiation successful
0924 21:09:31.957468 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1011,"mutex_wait_us":64,"thread_start_us":67,"threads_started":1}
I20260924 21:09:31.957556 12079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:09:31.941158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36278 (local address 127.0.98.2:40363)
0924 21:09:31.951611 (+ 10453us) server_negotiation.cc:206] Beginning negotiation
0924 21:09:31.951616 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 21:09:31.951628 (+ 12us) server_negotiation.cc:407] Connection header received
0924 21:09:31.951665 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:09:31.951667 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:09:31.951681 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:09:31.951772 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 21:09:31.951898 (+ 126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:09:31.952856 (+ 958us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:09:31.955174 (+ 2318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:09:31.956790 (+ 1616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:09:31.957353 (+ 563us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:09:31.957433 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0924 21:09:31.957456 (+ 23us) server_negotiation.cc:299] Negotiation successful
0924 21:09:31.957489 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10411,"thread_start_us":81,"threads_started":1}
I20260924 21:09:31.957926 12076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:09:31.940913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41004 (local address 127.0.98.3:35101)
0924 21:09:31.953892 (+ 12979us) server_negotiation.cc:206] Beginning negotiation
0924 21:09:31.953897 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 21:09:31.953907 (+ 10us) server_negotiation.cc:407] Connection header received
0924 21:09:31.953944 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:09:31.953946 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:09:31.953961 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:09:31.954093 (+ 132us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 21:09:31.954309 (+ 216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:09:31.955240 (+ 931us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:09:31.956572 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:09:31.957184 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:09:31.957245 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:09:31.957302 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0924 21:09:31.957821 (+ 519us) server_negotiation.cc:299] Negotiation successful
0924 21:09:31.957856 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12939,"thread_start_us":56,"threads_started":1}
I20260924 21:09:31.958099 11843 tablet_service.cc:1467] Processing CreateTablet for tablet 704b37e2f1bb4a458ac07a91acc760f5 (DEFAULT_TABLE table=test-table [id=a764df1a7fe047a0ac4823d4d698a5d7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 21:09:31.958391 11843 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 704b37e2f1bb4a458ac07a91acc760f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 21:09:31.961308 12083 tablet_bootstrap.cc:492] T 704b37e2f1bb4a458ac07a91acc760f5 P 1205036197a943938239309e0f2f4ce9: Bootstrap starting.
I20260924 21:09:31.961565 11996 tablet_service.cc:1467] Processing CreateTablet for tablet 704b37e2f1bb4a458ac07a91acc760f5 (DEFAULT_TABLE table=test-table [id=a764df1a7fe047a0ac4823d4d698a5d7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 21:09:31.961861 11996 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 704b37e2f1bb4a458ac07a91acc760f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 21:09:31.962200 12083 tablet_bootstrap.cc:654] T 704b37e2f1bb4a458ac07a91acc760f5 P 1205036197a943938239309e0f2f4ce9: Neither blocks nor log segments found. Creating new log.
I20260924 21:09:31.962456 12083 log.cc:824] T 704b37e2f1bb4a458ac07a91acc760f5 P 1205036197a943938239309e0f2f4ce9: Log is configured to *not* fsync() on all Append() calls
I20260924 21:09:31.963169 12081 tablet_bootstrap.cc:492] T 704b37e2f1bb4a458ac07a91acc760f5 P 07caf695041b44299e67de3255b71eda: No bootstrap required, opened a new log
I20260924 21:09:31.963239 12081 ts_tablet_manager.cc:1397] T 704b37e2f1bb4a458ac07a91acc760f5 P 07caf695041b44299e67de3255b71eda: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.000s
I20260924 21:09:31.965984 12081 raft_consensus.cc:348] T 704b37e2f1bb4a458ac07a91acc760f5 P 07caf695041b44299e67de3255b71eda [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1205036197a943938239309e0f2f4ce9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40363 } } peers { permanent_uuid: "07caf695041b44299e67de3255b71eda" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36413 } } peers { permanent_uuid: "f5b5f663ef0443518d8025f0ef3e75ae" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35101 } }
I20260924 21:09:31.966248 12081 raft_consensus.cc:374] T 704b37e2f1bb4a458ac07a91acc760f5 P 07caf695041b44299e67de3255b71eda [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 21:09:31.966324 12081 raft_consensus.cc:729] T 704b37e2f1bb4a458ac07a91acc760f5 P 07caf695041b44299e67de3255b71eda [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 07caf695041b44299e67de3255b71eda, State: Initialized, Role: FOLLOWER
I20260924 21:09:31.966506 12081 consensus_queue.cc:260] T 704b37e2f1bb4a458ac07a91acc760f5 P 07caf695041b44299e67de3255b71eda [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1205036197a943938239309e0f2f4ce9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40363 } } peers { permanent_uuid: "07caf695041b44299e67de3255b71eda" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36413 } } peers { permanent_uuid: "f5b5f663ef0443518d8025f0ef3e75ae" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35101 } }
I20260924 21:09:31.966835 12081 ts_tablet_manager.cc:1428] T 704b37e2f1bb4a458ac07a91acc760f5 P 07caf695041b44299e67de3255b71eda: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260924 21:09:31.966939 11790 heartbeater.cc:503] Master 127.0.98.62:46175 was elected leader, sending a full tablet report...
I20260924 21:09:31.967396 12085 tablet_bootstrap.cc:492] T 704b37e2f1bb4a458ac07a91acc760f5 P f5b5f663ef0443518d8025f0ef3e75ae: Bootstrap starting.
I20260924 21:09:31.968210 12085 tablet_bootstrap.cc:654] T 704b37e2f1bb4a458ac07a91acc760f5 P f5b5f663ef0443518d8025f0ef3e75ae: Neither blocks nor log segments found. Creating new log.
I20260924 21:09:31.968461 12085 log.cc:824] T 704b37e2f1bb4a458ac07a91acc760f5 P f5b5f663ef0443518d8025f0ef3e75ae: Log is configured to *not* fsync() on all Append() calls
I20260924 21:09:31.969153 12083 tablet_bootstrap.cc:492] T 704b37e2f1bb4a458ac07a91acc760f5 P 1205036197a943938239309e0f2f4ce9: No bootstrap required, opened a new log
I20260924 21:09:31.969252 12083 ts_tablet_manager.cc:1397] T 704b37e2f1bb4a458ac07a91acc760f5 P 1205036197a943938239309e0f2f4ce9: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20260924 21:09:31.969295 12085 tablet_bootstrap.cc:492] T 704b37e2f1bb4a458ac07a91acc760f5 P f5b5f663ef0443518d8025f0ef3e75ae: No bootstrap required, opened a new log
I20260924 21:09:31.969405 12085 ts_tablet_manager.cc:1397] T 704b37e2f1bb4a458ac07a91acc760f5 P f5b5f663ef0443518d8025f0ef3e75ae: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 21:09:31.970798 12083 raft_consensus.cc:348] T 704b37e2f1bb4a458ac07a91acc760f5 P 1205036197a943938239309e0f2f4ce9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1205036197a943938239309e0f2f4ce9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40363 } } peers { permanent_uuid: "07caf695041b44299e67de3255b71eda" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36413 } } peers { permanent_uuid: "f5b5f663ef0443518d8025f0ef3e75ae" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35101 } }
I20260924 21:09:31.970971 12083 raft_consensus.cc:374] T 704b37e2f1bb4a458ac07a91acc760f5 P 1205036197a943938239309e0f2f4ce9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 21:09:31.971045 12083 raft_consensus.cc:729] T 704b37e2f1bb4a458ac07a91acc760f5 P 1205036197a943938239309e0f2f4ce9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1205036197a943938239309e0f2f4ce9, State: Initialized, Role: FOLLOWER
I20260924 21:09:31.970984 12085 raft_consensus.cc:348] T 704b37e2f1bb4a458ac07a91acc760f5 P f5b5f663ef0443518d8025f0ef3e75ae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1205036197a943938239309e0f2f4ce9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40363 } } peers { permanent_uuid: "07caf695041b44299e67de3255b71eda" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36413 } } peers { permanent_uuid: "f5b5f663ef0443518d8025f0ef3e75ae" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35101 } }
I20260924 21:09:31.971180 12085 raft_consensus.cc:374] T 704b37e2f1bb4a458ac07a91acc760f5 P f5b5f663ef0443518d8025f0ef3e75ae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 21:09:31.971195 12083 consensus_queue.cc:260] T 704b37e2f1bb4a458ac07a91acc760f5 P 1205036197a943938239309e0f2f4ce9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1205036197a943938239309e0f2f4ce9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40363 } } peers { permanent_uuid: "07caf695041b44299e67de3255b71eda" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36413 } } peers { permanent_uuid: "f5b5f663ef0443518d8025f0ef3e75ae" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35101 } }
I20260924 21:09:31.971244 12085 raft_consensus.cc:729] T 704b37e2f1bb4a458ac07a91acc760f5 P f5b5f663ef0443518d8025f0ef3e75ae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f5b5f663ef0443518d8025f0ef3e75ae, State: Initialized, Role: FOLLOWER
I20260924 21:09:31.971400 12085 consensus_queue.cc:260] T 704b37e2f1bb4a458ac07a91acc760f5 P f5b5f663ef0443518d8025f0ef3e75ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1205036197a943938239309e0f2f4ce9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40363 } } peers { permanent_uuid: "07caf695041b44299e67de3255b71eda" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36413 } } peers { permanent_uuid: "f5b5f663ef0443518d8025f0ef3e75ae" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35101 } }
I20260924 21:09:31.971498 12083 ts_tablet_manager.cc:1428] T 704b37e2f1bb4a458ac07a91acc760f5 P 1205036197a943938239309e0f2f4ce9: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 21:09:31.971618 11927 heartbeater.cc:503] Master 127.0.98.62:46175 was elected leader, sending a full tablet report...
I20260924 21:09:31.971783 12085 ts_tablet_manager.cc:1428] T 704b37e2f1bb4a458ac07a91acc760f5 P f5b5f663ef0443518d8025f0ef3e75ae: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 21:09:31.971872 12062 heartbeater.cc:503] Master 127.0.98.62:46175 was elected leader, sending a full tablet report...
Sep 24 21:09:32 dist-test-slave-c7tk krb5kdc[11102](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284171, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
W20260924 21:09:32.118762 11928 tablet.cc:2369] T 704b37e2f1bb4a458ac07a91acc760f5 P 1205036197a943938239309e0f2f4ce9: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 21:09:32.128098 12063 tablet.cc:2369] T 704b37e2f1bb4a458ac07a91acc760f5 P f5b5f663ef0443518d8025f0ef3e75ae: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 21:09:32.159209 11791 tablet.cc:2369] T 704b37e2f1bb4a458ac07a91acc760f5 P 07caf695041b44299e67de3255b71eda: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 21:09:32.248162 12088 raft_consensus.cc:482] T 704b37e2f1bb4a458ac07a91acc760f5 P 1205036197a943938239309e0f2f4ce9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 21:09:32.248296 12088 raft_consensus.cc:504] T 704b37e2f1bb4a458ac07a91acc760f5 P 1205036197a943938239309e0f2f4ce9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1205036197a943938239309e0f2f4ce9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40363 } } peers { permanent_uuid: "07caf695041b44299e67de3255b71eda" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36413 } } peers { permanent_uuid: "f5b5f663ef0443518d8025f0ef3e75ae" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35101 } }
I20260924 21:09:32.248651 12088 leader_election.cc:288] T 704b37e2f1bb4a458ac07a91acc760f5 P 1205036197a943938239309e0f2f4ce9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 07caf695041b44299e67de3255b71eda (127.0.98.1:36413), f5b5f663ef0443518d8025f0ef3e75ae (127.0.98.3:35101)
I20260924 21:09:32.254977 12080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:09:32.249057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:55617 (local address 127.0.98.1:36413)
0924 21:09:32.249097 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0924 21:09:32.249099 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 21:09:32.249601 (+ 502us) server_negotiation.cc:407] Connection header received
0924 21:09:32.249692 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:09:32.249693 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:09:32.249704 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:09:32.249729 (+ 25us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 21:09:32.249931 (+ 202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:09:32.250597 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:09:32.254122 (+ 3525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:09:32.254664 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:09:32.254705 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:09:32.254714 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0924 21:09:32.254886 (+ 172us) server_negotiation.cc:299] Negotiation successful
0924 21:09:32.254917 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 21:09:32.255343 12090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:09:32.248871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36413 (local address 127.0.98.2:55617)
0924 21:09:32.249534 (+ 663us) negotiation.cc:107] Waiting for socket to connect
0924 21:09:32.249547 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 21:09:32.249627 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 21:09:32.249786 (+ 159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 21:09:32.249787 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 21:09:32.249788 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 21:09:32.249866 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:09:32.249868 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:09:32.250699 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:09:32.250700 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:09:32.251885 (+ 1185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:09:32.251888 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:09:32.254765 (+ 2877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:09:32.254765 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:09:32.254822 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:09:32.254825 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 21:09:32.255240 (+ 415us) client_negotiation.cc:240] Negotiation successful
0924 21:09:32.255278 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":647,"thread_start_us":65,"threads_started":1}
I20260924 21:09:32.255769 11744 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "704b37e2f1bb4a458ac07a91acc760f5" candidate_uuid: "1205036197a943938239309e0f2f4ce9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "07caf695041b44299e67de3255b71eda" is_pre_election: true
I20260924 21:09:32.255914 11744 raft_consensus.cc:2454] T 704b37e2f1bb4a458ac07a91acc760f5 P 07caf695041b44299e67de3255b71eda [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1205036197a943938239309e0f2f4ce9 in term 0.
I20260924 21:09:32.256213 11816 leader_election.cc:302] T 704b37e2f1bb4a458ac07a91acc760f5 P 1205036197a943938239309e0f2f4ce9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 07caf695041b44299e67de3255b71eda, 1205036197a943938239309e0f2f4ce9; no voters:
I20260924 21:09:32.256348 12088 raft_consensus.cc:2781] T 704b37e2f1bb4a458ac07a91acc760f5 P 1205036197a943938239309e0f2f4ce9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 21:09:32.256395 12088 raft_consensus.cc:482] T 704b37e2f1bb4a458ac07a91acc760f5 P 1205036197a943938239309e0f2f4ce9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 21:09:32.256420 12088 raft_consensus.cc:3037] T 704b37e2f1bb4a458ac07a91acc760f5 P 1205036197a943938239309e0f2f4ce9 [term 0 FOLLOWER]: Advancing to term 1
I20260924 21:09:32.256641 12076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:09:32.249473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:41653 (local address 127.0.98.3:35101)
0924 21:09:32.249527 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0924 21:09:32.249529 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 21:09:32.252035 (+ 2506us) server_negotiation.cc:407] Connection header received
0924 21:09:32.252115 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:09:32.252116 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:09:32.252126 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:09:32.252150 (+ 24us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 21:09:32.252309 (+ 159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:09:32.252876 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:09:32.255713 (+ 2837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:09:32.256227 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:09:32.256270 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:09:32.256282 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0924 21:09:32.256509 (+ 227us) server_negotiation.cc:299] Negotiation successful
0924 21:09:32.256555 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260924 21:09:32.256830 12091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:09:32.249343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:35101 (local address 127.0.98.2:41653)
0924 21:09:32.251978 (+ 2635us) negotiation.cc:107] Waiting for socket to connect
0924 21:09:32.251986 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0924 21:09:32.252057 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 21:09:32.252205 (+ 148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 21:09:32.252206 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 21:09:32.252207 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 21:09:32.252254 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:09:32.252255 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:09:32.252968 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:09:32.252969 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:09:32.254024 (+ 1055us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:09:32.254027 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:09:32.256338 (+ 2311us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:09:32.256339 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:09:32.256422 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:09:32.256426 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 21:09:32.256728 (+ 302us) client_negotiation.cc:240] Negotiation successful
0924 21:09:32.256760 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2623,"thread_start_us":48,"threads_started":1}
I20260924 21:09:32.257128 12016 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "704b37e2f1bb4a458ac07a91acc760f5" candidate_uuid: "1205036197a943938239309e0f2f4ce9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f5b5f663ef0443518d8025f0ef3e75ae" is_pre_election: true
I20260924 21:09:32.257253 12016 raft_consensus.cc:2454] T 704b37e2f1bb4a458ac07a91acc760f5 P f5b5f663ef0443518d8025f0ef3e75ae [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1205036197a943938239309e0f2f4ce9 in term 0.
I20260924 21:09:32.257330 12088 raft_consensus.cc:504] T 704b37e2f1bb4a458ac07a91acc760f5 P 1205036197a943938239309e0f2f4ce9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1205036197a943938239309e0f2f4ce9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40363 } } peers { permanent_uuid: "07caf695041b44299e67de3255b71eda" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36413 } } peers { permanent_uuid: "f5b5f663ef0443518d8025f0ef3e75ae" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35101 } }
I20260924 21:09:32.257448 12088 leader_election.cc:288] T 704b37e2f1bb4a458ac07a91acc760f5 P 1205036197a943938239309e0f2f4ce9 [CANDIDATE]: Term 1 election: Requested vote from peers 07caf695041b44299e67de3255b71eda (127.0.98.1:36413), f5b5f663ef0443518d8025f0ef3e75ae (127.0.98.3:35101)
I20260924 21:09:32.257947 12016 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "704b37e2f1bb4a458ac07a91acc760f5" candidate_uuid: "1205036197a943938239309e0f2f4ce9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f5b5f663ef0443518d8025f0ef3e75ae"
I20260924 21:09:32.258011 12016 raft_consensus.cc:3037] T 704b37e2f1bb4a458ac07a91acc760f5 P f5b5f663ef0443518d8025f0ef3e75ae [term 0 FOLLOWER]: Advancing to term 1
I20260924 21:09:32.258682 12016 raft_consensus.cc:2454] T 704b37e2f1bb4a458ac07a91acc760f5 P f5b5f663ef0443518d8025f0ef3e75ae [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1205036197a943938239309e0f2f4ce9 in term 1.
I20260924 21:09:32.258823 11817 leader_election.cc:302] T 704b37e2f1bb4a458ac07a91acc760f5 P 1205036197a943938239309e0f2f4ce9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1205036197a943938239309e0f2f4ce9, f5b5f663ef0443518d8025f0ef3e75ae; no voters:
I20260924 21:09:32.258955 11744 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "704b37e2f1bb4a458ac07a91acc760f5" candidate_uuid: "1205036197a943938239309e0f2f4ce9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "07caf695041b44299e67de3255b71eda"
I20260924 21:09:32.259016 11744 raft_consensus.cc:3037] T 704b37e2f1bb4a458ac07a91acc760f5 P 07caf695041b44299e67de3255b71eda [term 0 FOLLOWER]: Advancing to term 1
I20260924 21:09:32.259690 11744 raft_consensus.cc:2454] T 704b37e2f1bb4a458ac07a91acc760f5 P 07caf695041b44299e67de3255b71eda [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1205036197a943938239309e0f2f4ce9 in term 1.
I20260924 21:09:32.259894 12088 raft_consensus.cc:2781] T 704b37e2f1bb4a458ac07a91acc760f5 P 1205036197a943938239309e0f2f4ce9 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 21:09:32.260005 12088 raft_consensus.cc:686] T 704b37e2f1bb4a458ac07a91acc760f5 P 1205036197a943938239309e0f2f4ce9 [term 1 LEADER]: Becoming Leader. State: Replica: 1205036197a943938239309e0f2f4ce9, State: Running, Role: LEADER
I20260924 21:09:32.260074 12088 consensus_queue.cc:237] T 704b37e2f1bb4a458ac07a91acc760f5 P 1205036197a943938239309e0f2f4ce9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1205036197a943938239309e0f2f4ce9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40363 } } peers { permanent_uuid: "07caf695041b44299e67de3255b71eda" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36413 } } peers { permanent_uuid: "f5b5f663ef0443518d8025f0ef3e75ae" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35101 } }
I20260924 21:09:32.260772 11577 catalog_manager.cc:5495] T 704b37e2f1bb4a458ac07a91acc760f5 P 1205036197a943938239309e0f2f4ce9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1205036197a943938239309e0f2f4ce9 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "1205036197a943938239309e0f2f4ce9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1205036197a943938239309e0f2f4ce9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40363 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "07caf695041b44299e67de3255b71eda" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36413 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f5b5f663ef0443518d8025f0ef3e75ae" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35101 } health_report { overall_health: UNKNOWN } } }
I20260924 21:09:32.301108 12079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:09:32.298268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36288 (local address 127.0.98.2:40363)
0924 21:09:32.298305 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0924 21:09:32.298307 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 21:09:32.298313 (+ 6us) server_negotiation.cc:407] Connection header received
0924 21:09:32.298347 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:09:32.298348 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:09:32.298366 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:09:32.298421 (+ 55us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 21:09:32.298601 (+ 180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:09:32.299211 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:09:32.300158 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:09:32.300405 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:09:32.300456 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:09:32.300641 (+ 185us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 21:09:32.300755 (+ 114us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 21:09:32.300819 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0924 21:09:32.301002 (+ 183us) server_negotiation.cc:299] Negotiation successful
0924 21:09:32.301018 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 21:09:32.307309 12016 raft_consensus.cc:1261] T 704b37e2f1bb4a458ac07a91acc760f5 P f5b5f663ef0443518d8025f0ef3e75ae [term 1 FOLLOWER]: Refusing update from remote peer 1205036197a943938239309e0f2f4ce9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 21:09:32.307739 12092 consensus_queue.cc:1035] T 704b37e2f1bb4a458ac07a91acc760f5 P 1205036197a943938239309e0f2f4ce9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f5b5f663ef0443518d8025f0ef3e75ae" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 21:09:32.309726 11744 raft_consensus.cc:1261] T 704b37e2f1bb4a458ac07a91acc760f5 P 07caf695041b44299e67de3255b71eda [term 1 FOLLOWER]: Refusing update from remote peer 1205036197a943938239309e0f2f4ce9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 21:09:32.310072 12092 consensus_queue.cc:1035] T 704b37e2f1bb4a458ac07a91acc760f5 P 1205036197a943938239309e0f2f4ce9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "07caf695041b44299e67de3255b71eda" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36413 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Sep 24 21:09:32 dist-test-slave-c7tk krb5kdc[11102](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284171, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260924 21:09:32.313735 12096 mvcc.cc:205] Tried to move back new op lower bound from 7333003969767862272 to 7333003969577660416. Current Snapshot: MvccSnapshot[applied={T|T < 7333003969767862272}]
I20260924 21:09:32.313933 12099 mvcc.cc:205] Tried to move back new op lower bound from 7333003969767862272 to 7333003969577660416. Current Snapshot: MvccSnapshot[applied={T|T < 7333003969767862272}]
I20260924 21:09:32.315523 12097 mvcc.cc:205] Tried to move back new op lower bound from 7333003969767862272 to 7333003969577660416. Current Snapshot: MvccSnapshot[applied={T|T < 7333003969767862272}]
I20260924 21:09:32.318015 11577 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58616:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 21:09:32.318094 11577 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58616:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 21:09:32.318737 11577 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0d641ffbd17441b59f7ba554c14dc6d0: Sending DeleteTablet for 3 replicas of tablet 704b37e2f1bb4a458ac07a91acc760f5
I20260924 21:09:32.319072 11996 tablet_service.cc:1514] Processing DeleteTablet for tablet 704b37e2f1bb4a458ac07a91acc760f5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 21:09:32 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:41004
I20260924 21:09:32.319136 11724 tablet_service.cc:1514] Processing DeleteTablet for tablet 704b37e2f1bb4a458ac07a91acc760f5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 21:09:32 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:50446
I20260924 21:09:32.319329 12106 tablet_replica.cc:330] T 704b37e2f1bb4a458ac07a91acc760f5 P 07caf695041b44299e67de3255b71eda: stopping tablet replica
I20260924 21:09:32.319417 11843 tablet_service.cc:1514] Processing DeleteTablet for tablet 704b37e2f1bb4a458ac07a91acc760f5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 21:09:32 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:36278
I20260924 21:09:32.319445 12106 raft_consensus.cc:2229] T 704b37e2f1bb4a458ac07a91acc760f5 P 07caf695041b44299e67de3255b71eda [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 21:09:32.319545 12106 raft_consensus.cc:2258] T 704b37e2f1bb4a458ac07a91acc760f5 P 07caf695041b44299e67de3255b71eda [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 21:09:32.319953 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAzNNjS/build/release/bin/kudu with pid 11659
I20260924 21:09:32.326220 12107 tablet_replica.cc:330] T 704b37e2f1bb4a458ac07a91acc760f5 P 1205036197a943938239309e0f2f4ce9: stopping tablet replica
I20260924 21:09:32.326411 12107 raft_consensus.cc:2229] T 704b37e2f1bb4a458ac07a91acc760f5 P 1205036197a943938239309e0f2f4ce9 [term 1 LEADER]: Raft consensus shutting down.
I20260924 21:09:32.327643 12107 raft_consensus.cc:2258] T 704b37e2f1bb4a458ac07a91acc760f5 P 1205036197a943938239309e0f2f4ce9 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260924 21:09:32.327967 11564 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:36413 (error 108)
I20260924 21:09:32.328759 12078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:09:32.328662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36413 (local address 127.0.0.1:50452)
0924 21:09:32.328702 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0924 21:09:32.328728 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:36413: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
W20260924 21:09:32.328903 11564 catalog_manager.cc:4584] TS 07caf695041b44299e67de3255b71eda (127.0.98.1:36413): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 704b37e2f1bb4a458ac07a91acc760f5: Network error: Client connection negotiation failed: client connection to 127.0.98.1:36413: connect: Connection refused (error 111)
I20260924 21:09:32.329063 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAzNNjS/build/release/bin/kudu with pid 11796
I20260924 21:09:32.330579 12105 tablet_replica.cc:330] T 704b37e2f1bb4a458ac07a91acc760f5 P f5b5f663ef0443518d8025f0ef3e75ae: stopping tablet replica
I20260924 21:09:32.330742 12105 raft_consensus.cc:2229] T 704b37e2f1bb4a458ac07a91acc760f5 P f5b5f663ef0443518d8025f0ef3e75ae [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 21:09:32.330897 12105 raft_consensus.cc:2258] T 704b37e2f1bb4a458ac07a91acc760f5 P f5b5f663ef0443518d8025f0ef3e75ae [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 21:09:32.332646 12105 ts_tablet_manager.cc:1905] T 704b37e2f1bb4a458ac07a91acc760f5 P f5b5f663ef0443518d8025f0ef3e75ae: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260924 21:09:32.334860 12078 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 21:09:32.334618 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40363 (local address 127.0.0.1:36294)
0924 21:09:32.334657 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0924 21:09:32.334666 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 21:09:32.334699 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 21:09:32.334819 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:40363: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":21}
I20260924 21:09:32.334848 12105 ts_tablet_manager.cc:1918] T 704b37e2f1bb4a458ac07a91acc760f5 P f5b5f663ef0443518d8025f0ef3e75ae: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 21:09:32.334914 12105 log.cc:1196] T 704b37e2f1bb4a458ac07a91acc760f5 P f5b5f663ef0443518d8025f0ef3e75ae: Deleting WAL directory at /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790284076800039-392-0/minicluster-data/ts-2/wal/wals/704b37e2f1bb4a458ac07a91acc760f5
I20260924 21:09:32.335137 12105 ts_tablet_manager.cc:1939] T 704b37e2f1bb4a458ac07a91acc760f5 P f5b5f663ef0443518d8025f0ef3e75ae: Deleting consensus metadata
I20260924 21:09:32.335291 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAzNNjS/build/release/bin/kudu with pid 11933
W20260924 21:09:32.342701 12078 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 21:09:32.342427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:35101 (local address 127.0.0.1:41012)
0924 21:09:32.342534 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0924 21:09:32.342544 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 21:09:32.342577 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 21:09:32.342665 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.3:35101: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":92}
I20260924 21:09:32.343180 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAzNNjS/build/release/bin/kudu with pid 11546
I20260924 21:09:32.353997 392 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260924 21:09:32.697983 392 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260924 21:09:32.698072 392 mini_ranger.cc:66] Stopping Ranger...
I20260924 21:09:33.041864 392 mini_ranger.cc:68] Stopped Ranger
2026-09-24 21:09:33.042 UTC [11132] LOG: received smart shutdown request
2026-09-24 21:09:33.044 UTC [11132] LOG: background worker "logical replication launcher" (PID 11141) exited with exit code 1
2026-09-24 21:09:33.045 UTC [11136] LOG: shutting down
2026-09-24 21:09:33.213 UTC [11132] LOG: database system is shut down
2026-09-24T21:09:33Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (40915 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790284076800039-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 21:09:33 dist-test-slave-c7tk krb5kdc[12118](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 21:09:33 dist-test-slave-c7tk krb5kdc[12118](info): set up 2 sockets
Sep 24 21:09:33 dist-test-slave-c7tk krb5kdc[12118](info): commencing operation
krb5kdc: starting...
W20260924 21:09:34.237694 392 mini_kdc.cc:121] Time spent starting KDC: real 0.905s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790284076800039-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790284076800039-392-0/krb5kdc/test-user.keytab.
Sep 24 21:09:34 dist-test-slave-c7tk krb5kdc[12118](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284174, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T21:09:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T21:09:34Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790284076800039-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790284076800039-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790284076800039-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790284076800039-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790284076800039-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790284076800039-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
I20260924 21:09:34.375970 392 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790284076800039-392-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskAzNNjS/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790284076800039-392-0/minicluster-data/postgres -l logfile start
2026-09-24 21:09:36.533 UTC [12148] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-24 21:09:36.533 UTC [12148] LOG: listening on IPv4 address "127.0.98.20", port 45407
2026-09-24 21:09:36.536 UTC [12148] LOG: listening on Unix socket "/tmp/.s.PGSQL.45407"
2026-09-24 21:09:36.569 UTC [12151] LOG: database system was shut down at 2026-09-24 21:09:35 UTC
2026-09-24 21:09:36.573 UTC [12148] LOG: database system is ready to accept connections
I20260924 21:09:37.425064 392 mini_postgres.cc:96] Postgres bound to 45407
2026-09-24 21:09:37.430 UTC [12159] FATAL: database "slave" does not exist
127.0.98.20:45407 - accepting connections
I20260924 21:09:37.430959 392 mini_ranger.cc:161] Starting Ranger...
I20260924 21:09:37.444000 392 mini_ranger.cc:84] Created miniranger Postgres user
I20260924 21:09:37.835525 392 mini_ranger.cc:87] Created ranger Postgres database
I20260924 21:09:37.835606 392 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790284076800039-392-0/minicluster-data/ranger-admin
2026-09-24 21:09:38,086 [I] DB FLAVOR :POSTGRES
2026-09-24 21:09:38,086 [I] --------- Verifying Ranger DB connection ---------
2026-09-24 21:09:38,086 [I] Checking connection..
2026-09-24 21:09:38,086 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAzNNjS/build/release/bin/postgresql.jar:/tmp/dist-test-taskAzNNjS/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:45407/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-24 21:09:38,285 [I] Checking connection passed.
2026-09-24 21:09:38,285 [I] --------- Verifying version history table ---------
2026-09-24 21:09:38,286 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAzNNjS/build/release/bin/postgresql.jar:/tmp/dist-test-taskAzNNjS/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:45407/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-24 21:09:38,508 [I] Table x_db_version_h does not exist in database ranger
2026-09-24 21:09:38,508 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAzNNjS/build/release/bin/postgresql.jar:/tmp/dist-test-taskAzNNjS/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:45407/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-24 21:09:38,719 [I] Table x_db_version_h does not exist in database ranger
2026-09-24 21:09:38,719 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-24 21:09:38,719 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAzNNjS/build/release/bin/postgresql.jar:/tmp/dist-test-taskAzNNjS/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:45407/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskAzNNjS/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-24 21:09:38.942 UTC [12223] WARNING: there is no transaction in progress
2026-09-24 21:09:38,948 [I] create_dbversion_catalog.sql file imported successfully
2026-09-24 21:09:38,948 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAzNNjS/build/release/bin/postgresql.jar:/tmp/dist-test-taskAzNNjS/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:45407/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-24 21:09:39,173 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-24 21:09:39,173 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-24 21:09:39,173 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAzNNjS/build/release/bin/postgresql.jar:/tmp/dist-test-taskAzNNjS/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:45407/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-24 21:09:39,371 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAzNNjS/build/release/bin/postgresql.jar:/tmp/dist-test-taskAzNNjS/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:45407/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-24 21:09:39,590 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAzNNjS/build/release/bin/postgresql.jar:/tmp/dist-test-taskAzNNjS/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:45407/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-c7tk.c.gcp-upstream.internal','N') ;"
2026-09-24 21:09:39,810 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAzNNjS/build/release/bin/postgresql.jar:/tmp/dist-test-taskAzNNjS/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:45407/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-24 21:09:40,015 [I] Table x_portal_user does not exist in database ranger
2026-09-24 21:09:40,016 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAzNNjS/build/release/bin/postgresql.jar:/tmp/dist-test-taskAzNNjS/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:45407/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-24 21:09:40,233 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-24 21:09:40,234 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAzNNjS/build/release/bin/postgresql.jar:/tmp/dist-test-taskAzNNjS/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:45407/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-24 21:09:40,460 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAzNNjS/build/release/bin/postgresql.jar:/tmp/dist-test-taskAzNNjS/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:45407/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-24 21:09:40,688 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-24 21:09:40,688 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAzNNjS/build/release/bin/postgresql.jar:/tmp/dist-test-taskAzNNjS/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:45407/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskAzNNjS/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-24 21:09:41.512 UTC [12331] WARNING: there is no transaction in progress
2026-09-24 21:09:41.526 UTC [12331] WARNING: there is no transaction in progress
2026-09-24 21:09:41.540 UTC [12331] WARNING: there is no transaction in progress
2026-09-24 21:09:41.553 UTC [12331] WARNING: there is no transaction in progress
2026-09-24 21:09:41.570 UTC [12331] WARNING: there is no transaction in progress
2026-09-24 21:09:41.585 UTC [12331] WARNING: there is no transaction in progress
2026-09-24 21:09:41.634 UTC [12331] WARNING: there is no transaction in progress
2026-09-24 21:09:41.640 UTC [12331] WARNING: there is no transaction in progress
2026-09-24 21:09:41.652 UTC [12331] WARNING: there is no transaction in progress
2026-09-24 21:09:41.661 UTC [12331] WARNING: there is no transaction in progress
2026-09-24 21:09:41.670 UTC [12331] WARNING: there is no transaction in progress
2026-09-24 21:09:41.682 UTC [12331] WARNING: there is no transaction in progress
2026-09-24 21:09:41.691 UTC [12331] WARNING: there is no transaction in progress
2026-09-24 21:09:41.696 UTC [12331] WARNING: there is no transaction in progress
2026-09-24 21:09:41.702 UTC [12331] WARNING: there is no transaction in progress
2026-09-24 21:09:41.707 UTC [12331] WARNING: there is no transaction in progress
2026-09-24 21:09:42,238 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-24 21:09:42,239 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAzNNjS/build/release/bin/postgresql.jar:/tmp/dist-test-taskAzNNjS/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:45407/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-24 21:09:42,446 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-24 21:09:42,446 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAzNNjS/build/release/bin/postgresql.jar:/tmp/dist-test-taskAzNNjS/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:45407/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-24 21:09:42,677 [I] Table x_portal_user already exists in database 'ranger'
2026-09-24 21:09:42,677 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAzNNjS/build/release/bin/postgresql.jar:/tmp/dist-test-taskAzNNjS/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:45407/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-24 21:09:42,877 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-24 21:09:42,878 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAzNNjS/build/release/bin/postgresql.jar:/tmp/dist-test-taskAzNNjS/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:45407/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-24 21:09:43,086 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAzNNjS/build/release/bin/postgresql.jar:/tmp/dist-test-taskAzNNjS/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:45407/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-24 21:09:43,314 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAzNNjS/build/release/bin/postgresql.jar:/tmp/dist-test-taskAzNNjS/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:45407/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-c7tk.c.gcp-upstream.internal';"
2026-09-24 21:09:43,526 [I] CORE_DB_SCHEMA import status has been updated
2026-09-24 21:09:43,527 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAzNNjS/build/release/bin/postgresql.jar:/tmp/dist-test-taskAzNNjS/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:45407/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-24 21:09:43,760 [I] DB_PATCHES have already been applied
I20260924 21:09:43.764991 392 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790284076800039-392-0/minicluster-data/ranger-admin:/tmp/dist-test-taskAzNNjS/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskAzNNjS/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskAzNNjS/build/release/bin/postgresql.jar:/tmp/dist-test-taskAzNNjS/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260924 21:09:43.765046 392 mini_ranger.cc:193] Using host: 127.0.98.20
I20260924 21:09:43.767769 392 mini_ranger.cc:227] Ranger admin URL: http://127.0.98.20:55373
Sep 24, 2026 9:09:44 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 24, 2026 9:09:44 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 24, 2026 9:09:44 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790284076800039-392-0/minicluster-data/ranger-admin/ews/webapp
Sep 24, 2026 9:09:44 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790284076800039-392-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 24, 2026 9:09:44 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790284076800039-392-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 24, 2026 9:09:44 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790284076800039-392-0/minicluster-data/ranger-admin/ews/webapp].
Sep 24, 2026 9:09:44 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260924 21:09:44.754642 392 mini_ranger.cc:160] Time spent starting Ranger: real 7.324s user 0.000s sys 0.007s
Sep 24, 2026 9:10:02 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 24, 2026 9:10:02 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 24, 2026 9:10:02 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 24, 2026 9:10:02 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 24, 2026 9:10:02 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 24, 2026 9:10:02 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 24, 2026 9:10:02 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 24, 2026 9:10:02 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 24, 2026 9:10:02 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 24, 2026 9:10:02 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 24, 2026 9:10:02 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 24, 2026 9:10:02 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 24, 2026 9:10:02 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 24, 2026 9:10:02 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 24, 2026 9:10:02 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 24, 2026 9:10:02 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 24, 2026 9:10:02 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 24, 2026 9:10:03 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260924 21:10:03.855477 392 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790284076800039-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790284076800039-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.98.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790284076800039-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790284076800039-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 24 21:10:03 dist-test-slave-c7tk krb5kdc[12118](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284203, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260924 21:10:03.930848 392 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260924 21:10:03.938803 392 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260924 21:10:04.697423 392 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260924 21:10:04.697500 392 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790284076800039-392-0/minicluster-data/ranger-kms
I20260924 21:10:04.697518 392 mini_ranger_kms.cc:227] Using postgres at 127.0.98.20:45407
2026-09-24 21:10:04,841 [I] DB FLAVOR :POSTGRES
2026-09-24 21:10:04,842 [I] --------- Verifying Ranger DB connection ---------
2026-09-24 21:10:04,842 [I] Checking connection
2026-09-24 21:10:04,842 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAzNNjS/build/release/bin/postgresql.jar:/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790284076800039-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:45407/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-24 21:10:05,042 [I] connection success
2026-09-24 21:10:05,042 [I] --------- Verifying Ranger DB tables ---------
2026-09-24 21:10:05,042 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-24 21:10:05,042 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAzNNjS/build/release/bin/postgresql.jar:/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790284076800039-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:45407/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-24 21:10:05,256 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-24 21:10:05,257 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-24 21:10:05,257 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-24 21:10:05,257 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAzNNjS/build/release/bin/postgresql.jar:/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790284076800039-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:45407/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790284076800039-392-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-24 21:10:05,481 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260924 21:10:05.878165 392 mini_ranger_kms.cc:310] Created kms service
I20260924 21:10:06.070943 392 mini_ranger_kms.cc:326] Created kudu user
I20260924 21:10:06.109555 392 mini_ranger_kms.cc:343] Created rangerkms user
Sep 24 21:10:06 dist-test-slave-c7tk krb5kdc[12118](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284203, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260924 21:10:06.429692 392 mini_ranger_kms.cc:384] Added ranger policy
I20260924 21:10:06.429821 392 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790284076800039-392-0/minicluster-data/ranger-kms:/tmp/dist-test-taskAzNNjS/build/release/bin/postgresql.jar:/tmp/dist-test-taskAzNNjS/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskAzNNjS/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260924 21:10:06.429842 392 mini_ranger_kms.cc:240] Using host: 127.0.98.20
I20260924 21:10:06.433365 392 mini_ranger_kms.cc:276] Ranger KMS PID: 12511
I20260924 21:10:06.433446 392 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.98.20:56145
Sep 24, 2026 9:10:07 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 24, 2026 9:10:07 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 24, 2026 9:10:07 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790284076800039-392-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 24, 2026 9:10:07 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790284076800039-392-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 24, 2026 9:10:07 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790284076800039-392-0/minicluster-data/ranger-kms/ews/webapp].
Sep 24, 2026 9:10:07 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Sep 24 21:10:08 dist-test-slave-c7tk krb5kdc[12118](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790284208, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 21:10:08.441040 392 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 4.510s user 0.008s sys 0.122s
I20260924 21:10:08.441139 392 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260924 21:10:08.441195 392 mini_ranger_kms.cc:401] 127.0.98.20:56145/kms/v1/keys
Sep 24 21:10:08 dist-test-slave-c7tk krb5kdc[12118](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284208, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 24 21:10:09 dist-test-slave-c7tk krb5kdc[12118](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284208, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 24 21:10:10 dist-test-slave-c7tk krb5kdc[12118](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284210, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790284076800039-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790284076800039-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 21:10:10.898798 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAzNNjS/build/release/bin/kudu
/tmp/dist-test-taskAzNNjS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790284076800039-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790284076800039-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790284076800039-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790284076800039-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36107
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790284076800039-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41027
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36107
--ranger_config_path=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790284076800039-392-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:56145/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790284076800039-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790284076800039-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790284076800039-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 21:10:10.969910 12561 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 21:10:10.970095 12561 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 21:10:10.970116 12561 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 21:10:10.971213 12561 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 21:10:10.971254 12561 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 21:10:10.971268 12561 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 21:10:10.971280 12561 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 21:10:10.971292 12561 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 21:10:10.972854 12561 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41027
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790284076800039-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790284076800039-392-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.98.20:56145/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36107
--ranger_config_path=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790284076800039-392-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790284076800039-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36107
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790284076800039-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12561
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790284076800039-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 19:49:06 UTC on ccycloud.quasar-vhqjoc.root.comops.site
build id 4670864
I20260924 21:10:10.973068 12561 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 21:10:10.973289 12561 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 21:10:10.978377 12566 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 21:10:10.978471 12567 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 21:10:10.978655 12569 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 21:10:10.978739 12561 server_base.cc:1034] running on GCE node
I20260924 21:10:10.978935 12561 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 21:10:10.979207 12561 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 21:10:10.980357 12561 hybrid_clock.cc:648] HybridClock initialized: now 1790284210980327 us; error 44 us; skew 500 ppm
Sep 24 21:10:10 dist-test-slave-c7tk krb5kdc[12118](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284210, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 21:10:10.987869 12561 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 21:10:10.988269 12561 webserver.cc:460] Webserver started at http://127.0.98.62:44411/ using document root <none> and password file <none>
I20260924 21:10:10.988457 12561 fs_manager.cc:359] Metadata directory not provided
I20260924 21:10:10.988507 12561 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 21:10:10.988644 12561 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 21:10:10 dist-test-slave-c7tk krb5kdc[12118](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284210, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260924 21:10:11.028098 12561 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790284076800039-392-0/minicluster-data/master-0/data/instance:
uuid: "264c9f6880cb4d77b01bdc78088464dc"
format_stamp: "Formatted at 2026-09-24 21:10:11 on dist-test-slave-c7tk"
server_key: "10fe6714b80237b6bf799b568ecc2e7c"
server_key_iv: "1ad8a7dd95716cecd3fd560d85f8d703"
server_key_version: "kuduclusterkey@0"
I20260924 21:10:11.028505 12561 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790284076800039-392-0/minicluster-data/master-0/wal/instance:
uuid: "264c9f6880cb4d77b01bdc78088464dc"
format_stamp: "Formatted at 2026-09-24 21:10:11 on dist-test-slave-c7tk"
server_key: "10fe6714b80237b6bf799b568ecc2e7c"
server_key_iv: "1ad8a7dd95716cecd3fd560d85f8d703"
server_key_version: "kuduclusterkey@0"
I20260924 21:10:11.030212 12561 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.004s
I20260924 21:10:11.040055 12576 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 21:10:11.040242 12561 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260924 21:10:11.040344 12561 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790284076800039-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790284076800039-392-0/minicluster-data/master-0/wal
uuid: "264c9f6880cb4d77b01bdc78088464dc"
format_stamp: "Formatted at 2026-09-24 21:10:11 on dist-test-slave-c7tk"
server_key: "10fe6714b80237b6bf799b568ecc2e7c"
server_key_iv: "1ad8a7dd95716cecd3fd560d85f8d703"
server_key_version: "kuduclusterkey@0"
I20260924 21:10:11.040413 12561 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790284076800039-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790284076800039-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790284076800039-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 21:10:11.058060 12561 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 21:10:11.059859 12561 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 21:10:11.060011 12561 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 21:10:11.064607 12561 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36107
I20260924 21:10:11.064632 12628 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36107 every 8 connection(s)
I20260924 21:10:11.065006 12561 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790284076800039-392-0/minicluster-data/master-0/data/info.pb
I20260924 21:10:11.065625 12629 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 21:10:11.067761 12629 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 264c9f6880cb4d77b01bdc78088464dc: Bootstrap starting.
I20260924 21:10:11.068320 12629 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 264c9f6880cb4d77b01bdc78088464dc: Neither blocks nor log segments found. Creating new log.
I20260924 21:10:11.068578 12629 log.cc:824] T 00000000000000000000000000000000 P 264c9f6880cb4d77b01bdc78088464dc: Log is configured to *not* fsync() on all Append() calls
I20260924 21:10:11.069178 12629 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 264c9f6880cb4d77b01bdc78088464dc: No bootstrap required, opened a new log
I20260924 21:10:11.071116 12629 raft_consensus.cc:348] T 00000000000000000000000000000000 P 264c9f6880cb4d77b01bdc78088464dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "264c9f6880cb4d77b01bdc78088464dc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36107 } }
I20260924 21:10:11.071260 12629 raft_consensus.cc:374] T 00000000000000000000000000000000 P 264c9f6880cb4d77b01bdc78088464dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 21:10:11.071317 12629 raft_consensus.cc:729] T 00000000000000000000000000000000 P 264c9f6880cb4d77b01bdc78088464dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 264c9f6880cb4d77b01bdc78088464dc, State: Initialized, Role: FOLLOWER
I20260924 21:10:11.071473 12629 consensus_queue.cc:260] T 00000000000000000000000000000000 P 264c9f6880cb4d77b01bdc78088464dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "264c9f6880cb4d77b01bdc78088464dc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36107 } }
I20260924 21:10:11.071559 12629 raft_consensus.cc:388] T 00000000000000000000000000000000 P 264c9f6880cb4d77b01bdc78088464dc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 21:10:11.071596 12629 raft_consensus.cc:482] T 00000000000000000000000000000000 P 264c9f6880cb4d77b01bdc78088464dc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 21:10:11.071636 12629 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 264c9f6880cb4d77b01bdc78088464dc [term 0 FOLLOWER]: Advancing to term 1
I20260924 21:10:11.072081 12629 raft_consensus.cc:504] T 00000000000000000000000000000000 P 264c9f6880cb4d77b01bdc78088464dc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "264c9f6880cb4d77b01bdc78088464dc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36107 } }
I20260924 21:10:11.072181 12629 leader_election.cc:302] T 00000000000000000000000000000000 P 264c9f6880cb4d77b01bdc78088464dc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 264c9f6880cb4d77b01bdc78088464dc; no voters:
I20260924 21:10:11.072337 12629 leader_election.cc:288] T 00000000000000000000000000000000 P 264c9f6880cb4d77b01bdc78088464dc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 21:10:11.072386 12632 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 264c9f6880cb4d77b01bdc78088464dc [term 1 FOLLOWER]: Leader election won for term 1
I20260924 21:10:11.072563 12629 sys_catalog.cc:564] T 00000000000000000000000000000000 P 264c9f6880cb4d77b01bdc78088464dc [sys.catalog]: configured and running, proceeding with master startup.
I20260924 21:10:11.072613 12632 raft_consensus.cc:686] T 00000000000000000000000000000000 P 264c9f6880cb4d77b01bdc78088464dc [term 1 LEADER]: Becoming Leader. State: Replica: 264c9f6880cb4d77b01bdc78088464dc, State: Running, Role: LEADER
I20260924 21:10:11.072707 12632 consensus_queue.cc:237] T 00000000000000000000000000000000 P 264c9f6880cb4d77b01bdc78088464dc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "264c9f6880cb4d77b01bdc78088464dc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36107 } }
I20260924 21:10:11.073050 12633 sys_catalog.cc:455] T 00000000000000000000000000000000 P 264c9f6880cb4d77b01bdc78088464dc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "264c9f6880cb4d77b01bdc78088464dc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "264c9f6880cb4d77b01bdc78088464dc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36107 } } }
I20260924 21:10:11.073128 12633 sys_catalog.cc:458] T 00000000000000000000000000000000 P 264c9f6880cb4d77b01bdc78088464dc [sys.catalog]: This master's current role is: LEADER
I20260924 21:10:11.073261 12634 sys_catalog.cc:455] T 00000000000000000000000000000000 P 264c9f6880cb4d77b01bdc78088464dc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 264c9f6880cb4d77b01bdc78088464dc. Latest consensus state: current_term: 1 leader_uuid: "264c9f6880cb4d77b01bdc78088464dc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "264c9f6880cb4d77b01bdc78088464dc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36107 } } }
I20260924 21:10:11.073298 12634 sys_catalog.cc:458] T 00000000000000000000000000000000 P 264c9f6880cb4d77b01bdc78088464dc [sys.catalog]: This master's current role is: LEADER
I20260924 21:10:11.073378 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAzNNjS/build/release/bin/kudu as pid 12561
I20260924 21:10:11.073467 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790284076800039-392-0/minicluster-data/master-0/wal/instance
I20260924 21:10:11.073671 392 external_mini_cluster.cc:1428] Setting key 3ad44d3e92281d9c9553b17ca4e60456
I20260924 21:10:11.073719 12641 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 21:10:11.074230 12641 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 21:10:11.074651 12629 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790284076800039-392-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
Sep 24 21:10:11 dist-test-slave-c7tk krb5kdc[12118](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284210, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 21:10:11.086050 12641 catalog_manager.cc:1347] Generated new cluster ID: 6fe3c2ce291b4b30ac8726576398e5e0
I20260924 21:10:11.086205 12641 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 21:10:11.087568 12649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:10:11.074292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50872 (local address 127.0.98.62:36107)
0924 21:10:11.074457 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0924 21:10:11.074466 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 21:10:11.074645 (+ 179us) server_negotiation.cc:407] Connection header received
0924 21:10:11.074851 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:10:11.074859 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:10:11.074915 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:10:11.075053 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:10:11.077106 (+ 2053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:10:11.078700 (+ 1594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:10:11.079732 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:10:11.080384 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:10:11.080781 (+ 397us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:10:11.082822 (+ 2041us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 21:10:11.082826 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 21:10:11.082826 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 21:10:11.082836 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0924 21:10:11.085689 (+ 2853us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:10:11.086083 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:10:11.086085 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:10:11.086085 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:10:11.086191 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:10:11.086387 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:10:11.086388 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:10:11.086388 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:10:11.086660 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 21:10:11.087169 (+ 509us) server_negotiation.cc:1035] Waiting for connection context
0924 21:10:11.087230 (+ 61us) server_negotiation.cc:299] Negotiation successful
0924 21:10:11.087357 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108,"thread_start_us":59,"threads_started":1}
I20260924 21:10:11.097296 12641 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 21:10:11.097965 12641 catalog_manager.cc:1504] Loading token signing keys...
I20260924 21:10:11.104866 12641 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 264c9f6880cb4d77b01bdc78088464dc: Generated new TSK 0
I20260924 21:10:11.105062 12641 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790284076800039-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790284076800039-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260924 21:10:11.772915 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAzNNjS/build/release/bin/kudu
/tmp/dist-test-taskAzNNjS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790284076800039-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790284076800039-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790284076800039-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790284076800039-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:56145/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36107
--keytab_file=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790284076800039-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41027
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790284076800039-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790284076800039-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790284076800039-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 21:10:11.847179 12675 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 21:10:11.847337 12675 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 21:10:11.847355 12675 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 21:10:11.848361 12675 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 21:10:11.848404 12675 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 21:10:11.848428 12675 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 21:10:11.849741 12675 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41027
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790284076800039-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790284076800039-392-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.98.20:56145/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790284076800039-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790284076800039-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36107
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12675
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790284076800039-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 19:49:06 UTC on ccycloud.quasar-vhqjoc.root.comops.site
build id 4670864
I20260924 21:10:11.850118 12675 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 21:10:11.850344 12675 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 21:10:11.854847 12683 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 21:10:11.854882 12681 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 21:10:11.854990 12675 server_base.cc:1034] running on GCE node
W20260924 21:10:11.854885 12680 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 21:10:11.855325 12675 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 21:10:11.855551 12675 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 21:10:11.856719 12675 hybrid_clock.cc:648] HybridClock initialized: now 1790284211856704 us; error 29 us; skew 500 ppm
Sep 24 21:10:11 dist-test-slave-c7tk krb5kdc[12118](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284211, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 21:10:11.859577 12675 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 21:10:11.860054 12675 webserver.cc:460] Webserver started at http://127.0.98.1:35689/ using document root <none> and password file <none>
I20260924 21:10:11.860281 12675 fs_manager.cc:359] Metadata directory not provided
I20260924 21:10:11.860368 12675 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 21:10:11.860500 12675 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 21:10:11 dist-test-slave-c7tk krb5kdc[12118](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284211, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260924 21:10:11.873972 12675 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790284076800039-392-0/minicluster-data/ts-0/data/instance:
uuid: "bc21df88e45348ef92e3b18c1c4dcdcc"
format_stamp: "Formatted at 2026-09-24 21:10:11 on dist-test-slave-c7tk"
server_key: "917e7ca237d4549e22a6882c943375ec"
server_key_iv: "2f1e5ec841d38bdcc08553e12f30fa24"
server_key_version: "kuduclusterkey@0"
I20260924 21:10:11.874449 12675 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790284076800039-392-0/minicluster-data/ts-0/wal/instance:
uuid: "bc21df88e45348ef92e3b18c1c4dcdcc"
format_stamp: "Formatted at 2026-09-24 21:10:11 on dist-test-slave-c7tk"
server_key: "917e7ca237d4549e22a6882c943375ec"
server_key_iv: "2f1e5ec841d38bdcc08553e12f30fa24"
server_key_version: "kuduclusterkey@0"
I20260924 21:10:11.875734 12675 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 21:10:11.885227 12690 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 21:10:11.885459 12675 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 21:10:11.885542 12675 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790284076800039-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790284076800039-392-0/minicluster-data/ts-0/wal
uuid: "bc21df88e45348ef92e3b18c1c4dcdcc"
format_stamp: "Formatted at 2026-09-24 21:10:11 on dist-test-slave-c7tk"
server_key: "917e7ca237d4549e22a6882c943375ec"
server_key_iv: "2f1e5ec841d38bdcc08553e12f30fa24"
server_key_version: "kuduclusterkey@0"
I20260924 21:10:11.885628 12675 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790284076800039-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790284076800039-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790284076800039-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 21:10:11.901567 12675 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 21:10:11.903689 12675 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 21:10:11.903831 12675 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 21:10:11.904066 12675 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 21:10:11.904368 12675 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 21:10:11.904423 12675 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 21:10:11.904486 12675 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 21:10:11.904517 12675 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 21:10:11.910187 12675 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:42427
I20260924 21:10:11.910526 12675 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790284076800039-392-0/minicluster-data/ts-0/data/info.pb
I20260924 21:10:11.914531 12805 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:42427 every 8 connection(s)
I20260924 21:10:11.917389 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAzNNjS/build/release/bin/kudu as pid 12675
I20260924 21:10:11.917477 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790284076800039-392-0/minicluster-data/ts-0/wal/instance
I20260924 21:10:11.917668 392 external_mini_cluster.cc:1428] Setting key bb5456881dfe7eb4088ca206be195fc6
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Sep 24 21:10:11 dist-test-slave-c7tk krb5kdc[12118](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284211, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 21:10:11.934381 12809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:10:11.915178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:57877 (local address 127.0.98.62:36107)
0924 21:10:11.915339 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0924 21:10:11.915341 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 21:10:11.922036 (+ 6695us) server_negotiation.cc:407] Connection header received
0924 21:10:11.922648 (+ 612us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:10:11.922649 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:10:11.922662 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:10:11.922703 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:10:11.923513 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:10:11.924295 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:10:11.925615 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:10:11.925900 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:10:11.925933 (+ 33us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:10:11.931091 (+ 5158us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 21:10:11.931093 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 21:10:11.931094 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 21:10:11.931101 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 21:10:11.932662 (+ 1561us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:10:11.933172 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:10:11.933173 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:10:11.933174 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:10:11.933240 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:10:11.933523 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:10:11.933524 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:10:11.933524 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:10:11.933660 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 21:10:11.933708 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0924 21:10:11.934140 (+ 432us) server_negotiation.cc:299] Negotiation successful
0924 21:10:11.934230 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":107,"threads_started":1}
I20260924 21:10:11.935088 12808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:10:11.915136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36107 (local address 127.0.98.1:57877)
0924 21:10:11.921915 (+ 6779us) negotiation.cc:107] Waiting for socket to connect
0924 21:10:11.921959 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0924 21:10:11.922538 (+ 579us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 21:10:11.922912 (+ 374us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 21:10:11.922916 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 21:10:11.923145 (+ 229us) client_negotiation.cc:189] Negotiated authn=SASL
0924 21:10:11.923408 (+ 263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:10:11.923413 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:10:11.924406 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:10:11.924407 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:10:11.925512 (+ 1105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:10:11.925514 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:10:11.926011 (+ 497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:10:11.926011 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:10:11.926111 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:10:11.926548 (+ 437us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 21:10:11.926555 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0924 21:10:11.930989 (+ 4434us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 21:10:11.932744 (+ 1755us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 21:10:11.932750 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 21:10:11.932751 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 21:10:11.933033 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 21:10:11.933313 (+ 280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 21:10:11.933314 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 21:10:11.933314 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 21:10:11.933446 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 21:10:11.933759 (+ 313us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 21:10:11.933765 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 21:10:11.933972 (+ 207us) client_negotiation.cc:769] Sending connection context
0924 21:10:11.934706 (+ 734us) client_negotiation.cc:240] Negotiation successful
0924 21:10:11.934879 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6738,"thread_start_us":79,"threads_started":1}
I20260924 21:10:11.935724 12806 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36107
I20260924 21:10:11.935798 12806 heartbeater.cc:464] Registering TS with master...
I20260924 21:10:11.935966 12806 heartbeater.cc:511] Master 127.0.98.62:36107 requested a full tablet report, sending...
I20260924 21:10:11.936480 12593 ts_manager.cc:194] Registered new tserver with Master: bc21df88e45348ef92e3b18c1c4dcdcc (127.0.98.1:42427)
I20260924 21:10:11.937273 12593 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:57877
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790284076800039-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790284076800039-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260924 21:10:11.944206 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAzNNjS/build/release/bin/kudu
/tmp/dist-test-taskAzNNjS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790284076800039-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790284076800039-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790284076800039-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790284076800039-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:56145/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36107
--keytab_file=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790284076800039-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41027
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790284076800039-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790284076800039-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790284076800039-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 21:10:12.038331 12812 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 21:10:12.038555 12812 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 21:10:12.038585 12812 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 21:10:12.040107 12812 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 21:10:12.040202 12812 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 21:10:12.040261 12812 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 21:10:12.042117 12812 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41027
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790284076800039-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790284076800039-392-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.98.20:56145/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790284076800039-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790284076800039-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36107
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12812
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790284076800039-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 19:49:06 UTC on ccycloud.quasar-vhqjoc.root.comops.site
build id 4670864
I20260924 21:10:12.042439 12812 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 21:10:12.042665 12812 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 21:10:12.048075 12820 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 21:10:12.048354 12822 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 21:10:12.048588 12819 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 21:10:12.049544 12812 server_base.cc:1034] running on GCE node
I20260924 21:10:12.049747 12812 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 21:10:12.050030 12812 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 21:10:12.051174 12812 hybrid_clock.cc:648] HybridClock initialized: now 1790284212051156 us; error 36 us; skew 500 ppm
Sep 24 21:10:12 dist-test-slave-c7tk krb5kdc[12118](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284212, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 21:10:12.053550 12812 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 21:10:12.053987 12812 webserver.cc:460] Webserver started at http://127.0.98.2:41933/ using document root <none> and password file <none>
I20260924 21:10:12.054157 12812 fs_manager.cc:359] Metadata directory not provided
I20260924 21:10:12.054185 12812 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 21:10:12.054258 12812 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 21:10:12 dist-test-slave-c7tk krb5kdc[12118](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284212, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 24 21:10:12 dist-test-slave-c7tk krb5kdc[12118](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790284212, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 21:10:12.066946 12812 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790284076800039-392-0/minicluster-data/ts-1/data/instance:
uuid: "487f4094f26047459a40e54292b6716d"
format_stamp: "Formatted at 2026-09-24 21:10:12 on dist-test-slave-c7tk"
server_key: "c03e3c6975e278fa3be2963a6dc2eaed"
server_key_iv: "3274653c8840e1adcadf1318992e170f"
server_key_version: "kuduclusterkey@0"
I20260924 21:10:12.067317 12812 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790284076800039-392-0/minicluster-data/ts-1/wal/instance:
uuid: "487f4094f26047459a40e54292b6716d"
format_stamp: "Formatted at 2026-09-24 21:10:12 on dist-test-slave-c7tk"
server_key: "c03e3c6975e278fa3be2963a6dc2eaed"
server_key_iv: "3274653c8840e1adcadf1318992e170f"
server_key_version: "kuduclusterkey@0"
I20260924 21:10:12.090215 12812 fs_manager.cc:691] Time spent creating directory manager: real 0.023s user 0.001s sys 0.000s
I20260924 21:10:12.104486 12829 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 21:10:12.106886 12812 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.001s sys 0.000s
I20260924 21:10:12.106979 12812 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790284076800039-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790284076800039-392-0/minicluster-data/ts-1/wal
uuid: "487f4094f26047459a40e54292b6716d"
format_stamp: "Formatted at 2026-09-24 21:10:12 on dist-test-slave-c7tk"
server_key: "c03e3c6975e278fa3be2963a6dc2eaed"
server_key_iv: "3274653c8840e1adcadf1318992e170f"
server_key_version: "kuduclusterkey@0"
I20260924 21:10:12.107084 12812 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790284076800039-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790284076800039-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790284076800039-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 21:10:12.129114 12812 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 21:10:12.131387 12812 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 21:10:12.131544 12812 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 21:10:12.131752 12812 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 21:10:12.132081 12812 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 21:10:12.132153 12812 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 21:10:12.132212 12812 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 21:10:12.132262 12812 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 21:10:12.137714 12812 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:34245
I20260924 21:10:12.138134 12812 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790284076800039-392-0/minicluster-data/ts-1/data/info.pb
I20260924 21:10:12.138597 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAzNNjS/build/release/bin/kudu as pid 12812
I20260924 21:10:12.138675 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790284076800039-392-0/minicluster-data/ts-1/wal/instance
I20260924 21:10:12.138828 392 external_mini_cluster.cc:1428] Setting key ea1416435fc852d011c8bc1047e8c0c7
I20260924 21:10:12.140756 12943 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:34245 every 8 connection(s)
Sep 24 21:10:12 dist-test-slave-c7tk krb5kdc[12118](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284212, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 21:10:12.150907 12809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:10:12.141570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:33745 (local address 127.0.98.62:36107)
0924 21:10:12.141612 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0924 21:10:12.141614 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 21:10:12.141864 (+ 250us) server_negotiation.cc:407] Connection header received
0924 21:10:12.142471 (+ 607us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:10:12.142472 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:10:12.142487 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:10:12.142514 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:10:12.143189 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:10:12.143945 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:10:12.145355 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:10:12.145638 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:10:12.145674 (+ 36us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:10:12.147825 (+ 2151us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 21:10:12.147827 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 21:10:12.147827 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 21:10:12.147834 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 21:10:12.149308 (+ 1474us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:10:12.149818 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:10:12.149819 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:10:12.149819 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:10:12.149906 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:10:12.150181 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:10:12.150182 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:10:12.150182 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:10:12.150304 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 21:10:12.150344 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0924 21:10:12.150743 (+ 399us) server_negotiation.cc:299] Negotiation successful
0924 21:10:12.150829 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 21:10:12.151522 12947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:10:12.141322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36107 (local address 127.0.98.2:33745)
0924 21:10:12.141747 (+ 425us) negotiation.cc:107] Waiting for socket to connect
0924 21:10:12.141782 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0924 21:10:12.142349 (+ 567us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 21:10:12.142670 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 21:10:12.142674 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 21:10:12.142864 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0924 21:10:12.143091 (+ 227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:10:12.143097 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:10:12.144068 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:10:12.144069 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:10:12.145274 (+ 1205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:10:12.145277 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:10:12.145755 (+ 478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:10:12.145756 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:10:12.145883 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:10:12.146323 (+ 440us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 21:10:12.146331 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 21:10:12.147727 (+ 1396us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 21:10:12.149408 (+ 1681us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 21:10:12.149430 (+ 22us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 21:10:12.149432 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 21:10:12.149724 (+ 292us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 21:10:12.149982 (+ 258us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 21:10:12.149983 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 21:10:12.149983 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 21:10:12.150106 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 21:10:12.150404 (+ 298us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 21:10:12.150409 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 21:10:12.150577 (+ 168us) client_negotiation.cc:769] Sending connection context
0924 21:10:12.151189 (+ 612us) client_negotiation.cc:240] Negotiation successful
0924 21:10:12.151319 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":389,"thread_start_us":97,"threads_started":1}
I20260924 21:10:12.152136 12944 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36107
I20260924 21:10:12.152212 12944 heartbeater.cc:464] Registering TS with master...
I20260924 21:10:12.152367 12944 heartbeater.cc:511] Master 127.0.98.62:36107 requested a full tablet report, sending...
I20260924 21:10:12.152698 12593 ts_manager.cc:194] Registered new tserver with Master: 487f4094f26047459a40e54292b6716d (127.0.98.2:34245)
I20260924 21:10:12.153127 12593 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:33745
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790284076800039-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790284076800039-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260924 21:10:12.178830 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAzNNjS/build/release/bin/kudu
/tmp/dist-test-taskAzNNjS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790284076800039-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790284076800039-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790284076800039-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790284076800039-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:56145/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36107
--keytab_file=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790284076800039-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41027
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790284076800039-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790284076800039-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790284076800039-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 21:10:12.284780 12949 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 21:10:12.285020 12949 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 21:10:12.285056 12949 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 21:10:12.286542 12949 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 21:10:12.286595 12949 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 21:10:12.286631 12949 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 21:10:12.288518 12949 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41027
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790284076800039-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790284076800039-392-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.98.20:56145/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790284076800039-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790284076800039-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36107
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12949
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790284076800039-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 19:49:06 UTC on ccycloud.quasar-vhqjoc.root.comops.site
build id 4670864
I20260924 21:10:12.288821 12949 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 21:10:12.289070 12949 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 21:10:12.296365 12957 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 21:10:12.296691 12955 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 21:10:12.296844 12954 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 21:10:12.297744 12949 server_base.cc:1034] running on GCE node
I20260924 21:10:12.297958 12949 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 21:10:12.298179 12949 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 21:10:12.299369 12949 hybrid_clock.cc:648] HybridClock initialized: now 1790284212299343 us; error 41 us; skew 500 ppm
Sep 24 21:10:12 dist-test-slave-c7tk krb5kdc[12118](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284212, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 21:10:12.301780 12949 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 21:10:12.302227 12949 webserver.cc:460] Webserver started at http://127.0.98.3:33993/ using document root <none> and password file <none>
I20260924 21:10:12.302426 12949 fs_manager.cc:359] Metadata directory not provided
I20260924 21:10:12.302485 12949 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 21:10:12.302605 12949 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 21:10:12 dist-test-slave-c7tk krb5kdc[12118](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284212, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260924 21:10:12.314355 12949 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790284076800039-392-0/minicluster-data/ts-2/data/instance:
uuid: "bf6df1fafc5149e9861501adbce69453"
format_stamp: "Formatted at 2026-09-24 21:10:12 on dist-test-slave-c7tk"
server_key: "4bc3d96635bdcacf961a9b4cf2bd569c"
server_key_iv: "46e04b1e36f770a8ee4b9d79c3af35e1"
server_key_version: "kuduclusterkey@0"
I20260924 21:10:12.314728 12949 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790284076800039-392-0/minicluster-data/ts-2/wal/instance:
uuid: "bf6df1fafc5149e9861501adbce69453"
format_stamp: "Formatted at 2026-09-24 21:10:12 on dist-test-slave-c7tk"
server_key: "4bc3d96635bdcacf961a9b4cf2bd569c"
server_key_iv: "46e04b1e36f770a8ee4b9d79c3af35e1"
server_key_version: "kuduclusterkey@0"
I20260924 21:10:12.315933 12949 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 21:10:12.329207 12964 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 21:10:12.329424 12949 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 21:10:12.329524 12949 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790284076800039-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790284076800039-392-0/minicluster-data/ts-2/wal
uuid: "bf6df1fafc5149e9861501adbce69453"
format_stamp: "Formatted at 2026-09-24 21:10:12 on dist-test-slave-c7tk"
server_key: "4bc3d96635bdcacf961a9b4cf2bd569c"
server_key_iv: "46e04b1e36f770a8ee4b9d79c3af35e1"
server_key_version: "kuduclusterkey@0"
I20260924 21:10:12.329600 12949 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790284076800039-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790284076800039-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790284076800039-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 21:10:12.343192 12949 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 21:10:12.345343 12949 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 21:10:12.345539 12949 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 21:10:12.345777 12949 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 21:10:12.346159 12949 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 21:10:12.346271 12949 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 21:10:12.346336 12949 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 21:10:12.346403 12949 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 21:10:12.352123 12949 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:41583
I20260924 21:10:12.352525 12949 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790284076800039-392-0/minicluster-data/ts-2/data/info.pb
I20260924 21:10:12.354514 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAzNNjS/build/release/bin/kudu as pid 12949
I20260924 21:10:12.354604 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790284076800039-392-0/minicluster-data/ts-2/wal/instance
I20260924 21:10:12.354763 392 external_mini_cluster.cc:1428] Setting key 61e9f34c1f97e0e5bc30b166d8977cb6
Sep 24 21:10:12 dist-test-slave-c7tk krb5kdc[12118](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284212, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 21:10:12.365020 12809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:10:12.353541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:46755 (local address 127.0.98.62:36107)
0924 21:10:12.353582 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0924 21:10:12.353584 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 21:10:12.353827 (+ 243us) server_negotiation.cc:407] Connection header received
0924 21:10:12.356250 (+ 2423us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:10:12.356251 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:10:12.356261 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:10:12.356289 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:10:12.357081 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:10:12.357945 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:10:12.359231 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:10:12.359522 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:10:12.359600 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:10:12.361968 (+ 2368us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 21:10:12.361970 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 21:10:12.361970 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 21:10:12.361978 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 21:10:12.363569 (+ 1591us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:10:12.364033 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:10:12.364034 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:10:12.364034 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:10:12.364104 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:10:12.364375 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:10:12.364375 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:10:12.364376 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:10:12.364495 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 21:10:12.364559 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0924 21:10:12.364851 (+ 292us) server_negotiation.cc:299] Negotiation successful
0924 21:10:12.364932 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 21:10:12.365173 13080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:10:12.353548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36107 (local address 127.0.98.3:46755)
0924 21:10:12.353689 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0924 21:10:12.353725 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0924 21:10:12.356141 (+ 2416us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 21:10:12.356500 (+ 359us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 21:10:12.356505 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 21:10:12.356735 (+ 230us) client_negotiation.cc:189] Negotiated authn=SASL
0924 21:10:12.356977 (+ 242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:10:12.356983 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:10:12.358054 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:10:12.358055 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:10:12.359138 (+ 1083us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:10:12.359141 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:10:12.359631 (+ 490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:10:12.359633 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:10:12.359774 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:10:12.360284 (+ 510us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 21:10:12.360293 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 21:10:12.361841 (+ 1548us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 21:10:12.363660 (+ 1819us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 21:10:12.363665 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 21:10:12.363667 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 21:10:12.363954 (+ 287us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 21:10:12.364177 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 21:10:12.364177 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 21:10:12.364178 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 21:10:12.364300 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 21:10:12.364568 (+ 268us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 21:10:12.364573 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 21:10:12.364719 (+ 146us) client_negotiation.cc:769] Sending connection context
0924 21:10:12.364830 (+ 111us) client_negotiation.cc:240] Negotiation successful
0924 21:10:12.364974 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":79,"threads_started":1}
I20260924 21:10:12.365828 13078 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36107
I20260924 21:10:12.365935 13078 heartbeater.cc:464] Registering TS with master...
I20260924 21:10:12.366107 13078 heartbeater.cc:511] Master 127.0.98.62:36107 requested a full tablet report, sending...
I20260924 21:10:12.366473 12593 ts_manager.cc:194] Registered new tserver with Master: bf6df1fafc5149e9861501adbce69453 (127.0.98.3:41583)
I20260924 21:10:12.366743 13077 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:41583 every 8 connection(s)
I20260924 21:10:12.366909 12593 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:46755
I20260924 21:10:12.376868 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 21:10:12.377547 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAzNNjS/build/release/bin/kudu with pid 12675
I20260924 21:10:12.383256 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAzNNjS/build/release/bin/kudu with pid 12812
I20260924 21:10:12.387593 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAzNNjS/build/release/bin/kudu with pid 12949
I20260924 21:10:12.391898 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAzNNjS/build/release/bin/kudu with pid 12561
I20260924 21:10:12.396457 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAzNNjS/build/release/bin/kudu
/tmp/dist-test-taskAzNNjS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790284076800039-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790284076800039-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790284076800039-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790284076800039-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36107
--webserver_interface=127.0.98.62
--webserver_port=44411
--keytab_file=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790284076800039-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41027
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36107
--ranger_config_path=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790284076800039-392-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:56145/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:56145/kms with env {KRB5CCNAME=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790284076800039-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790284076800039-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790284076800039-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 21:10:12.466172 13081 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 21:10:12.466326 13081 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 21:10:12.466344 13081 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 21:10:12.467232 13081 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 21:10:12.467260 13081 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 21:10:12.467272 13081 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 21:10:12.467283 13081 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 21:10:12.467293 13081 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 21:10:12.468513 13081 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41027
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790284076800039-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790284076800039-392-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:56145/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36107
--ranger_config_path=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790284076800039-392-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790284076800039-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36107
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790284076800039-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=44411
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13081
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790284076800039-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 19:49:06 UTC on ccycloud.quasar-vhqjoc.root.comops.site
build id 4670864
I20260924 21:10:12.468652 13081 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 21:10:12.468777 13081 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 21:10:12.472981 13086 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 21:10:12.472983 13089 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 21:10:12.473003 13087 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 21:10:12.473078 13081 server_base.cc:1034] running on GCE node
I20260924 21:10:12.473343 13081 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 21:10:12.473603 13081 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 21:10:12.474748 13081 hybrid_clock.cc:648] HybridClock initialized: now 1790284212474744 us; error 33 us; skew 500 ppm
Sep 24 21:10:12 dist-test-slave-c7tk krb5kdc[12118](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284212, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 21:10:12.476826 13081 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 21:10:12.477221 13081 webserver.cc:460] Webserver started at http://127.0.98.62:44411/ using document root <none> and password file <none>
I20260924 21:10:12.477392 13081 fs_manager.cc:359] Metadata directory not provided
I20260924 21:10:12.477458 13081 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 24 21:10:12 dist-test-slave-c7tk krb5kdc[12118](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284212, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260924 21:10:12.533576 13081 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 21:10:12.534483 13097 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 21:10:12.534659 13081 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260924 21:10:12.534742 13081 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790284076800039-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790284076800039-392-0/minicluster-data/master-0/wal
uuid: "264c9f6880cb4d77b01bdc78088464dc"
format_stamp: "Formatted at 2026-09-24 21:10:11 on dist-test-slave-c7tk"
server_key: "10fe6714b80237b6bf799b568ecc2e7c"
server_key_iv: "1ad8a7dd95716cecd3fd560d85f8d703"
server_key_version: "kuduclusterkey@0"
I20260924 21:10:12.535123 13081 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790284076800039-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790284076800039-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790284076800039-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 21:10:12.556074 13081 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 21:10:12.557451 13081 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 21:10:12.557572 13081 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 21:10:12.561458 13081 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36107
I20260924 21:10:12.561520 13149 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36107 every 8 connection(s)
I20260924 21:10:12.561950 13081 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790284076800039-392-0/minicluster-data/master-0/data/info.pb
I20260924 21:10:12.563055 13150 sys_catalog.cc:263] Verifying existing consensus state
I20260924 21:10:12.563685 13150 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 264c9f6880cb4d77b01bdc78088464dc: Bootstrap starting.
I20260924 21:10:12.570937 13150 log.cc:824] T 00000000000000000000000000000000 P 264c9f6880cb4d77b01bdc78088464dc: Log is configured to *not* fsync() on all Append() calls
I20260924 21:10:12.571168 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAzNNjS/build/release/bin/kudu as pid 13081
I20260924 21:10:12.571441 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAzNNjS/build/release/bin/kudu
/tmp/dist-test-taskAzNNjS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790284076800039-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790284076800039-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790284076800039-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790284076800039-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:56145/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:42427
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:36107
--webserver_port=35689
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790284076800039-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41027
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:56145/kms with env {KRB5CCNAME=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790284076800039-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790284076800039-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790284076800039-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 21:10:12.572794 13150 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 264c9f6880cb4d77b01bdc78088464dc: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 21:10:12.573033 13150 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 264c9f6880cb4d77b01bdc78088464dc: Bootstrap complete.
I20260924 21:10:12.575196 13150 raft_consensus.cc:348] T 00000000000000000000000000000000 P 264c9f6880cb4d77b01bdc78088464dc [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "264c9f6880cb4d77b01bdc78088464dc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36107 } }
I20260924 21:10:12.575402 13150 raft_consensus.cc:729] T 00000000000000000000000000000000 P 264c9f6880cb4d77b01bdc78088464dc [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 264c9f6880cb4d77b01bdc78088464dc, State: Initialized, Role: FOLLOWER
I20260924 21:10:12.575546 13150 consensus_queue.cc:260] T 00000000000000000000000000000000 P 264c9f6880cb4d77b01bdc78088464dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "264c9f6880cb4d77b01bdc78088464dc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36107 } }
I20260924 21:10:12.575604 13150 raft_consensus.cc:388] T 00000000000000000000000000000000 P 264c9f6880cb4d77b01bdc78088464dc [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 21:10:12.575627 13150 raft_consensus.cc:482] T 00000000000000000000000000000000 P 264c9f6880cb4d77b01bdc78088464dc [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 21:10:12.575685 13150 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 264c9f6880cb4d77b01bdc78088464dc [term 1 FOLLOWER]: Advancing to term 2
I20260924 21:10:12.576511 13150 raft_consensus.cc:504] T 00000000000000000000000000000000 P 264c9f6880cb4d77b01bdc78088464dc [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "264c9f6880cb4d77b01bdc78088464dc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36107 } }
I20260924 21:10:12.576584 13150 leader_election.cc:302] T 00000000000000000000000000000000 P 264c9f6880cb4d77b01bdc78088464dc [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 264c9f6880cb4d77b01bdc78088464dc; no voters:
I20260924 21:10:12.576756 13150 leader_election.cc:288] T 00000000000000000000000000000000 P 264c9f6880cb4d77b01bdc78088464dc [CANDIDATE]: Term 2 election: Requested vote from peers
I20260924 21:10:12.576787 13154 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 264c9f6880cb4d77b01bdc78088464dc [term 2 FOLLOWER]: Leader election won for term 2
I20260924 21:10:12.576877 13154 raft_consensus.cc:686] T 00000000000000000000000000000000 P 264c9f6880cb4d77b01bdc78088464dc [term 2 LEADER]: Becoming Leader. State: Replica: 264c9f6880cb4d77b01bdc78088464dc, State: Running, Role: LEADER
I20260924 21:10:12.576970 13150 sys_catalog.cc:564] T 00000000000000000000000000000000 P 264c9f6880cb4d77b01bdc78088464dc [sys.catalog]: configured and running, proceeding with master startup.
I20260924 21:10:12.576953 13154 consensus_queue.cc:237] T 00000000000000000000000000000000 P 264c9f6880cb4d77b01bdc78088464dc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "264c9f6880cb4d77b01bdc78088464dc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36107 } }
I20260924 21:10:12.577167 13155 sys_catalog.cc:455] T 00000000000000000000000000000000 P 264c9f6880cb4d77b01bdc78088464dc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 264c9f6880cb4d77b01bdc78088464dc. Latest consensus state: current_term: 2 leader_uuid: "264c9f6880cb4d77b01bdc78088464dc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "264c9f6880cb4d77b01bdc78088464dc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36107 } } }
I20260924 21:10:12.577368 13155 sys_catalog.cc:458] T 00000000000000000000000000000000 P 264c9f6880cb4d77b01bdc78088464dc [sys.catalog]: This master's current role is: LEADER
I20260924 21:10:12.577318 13156 sys_catalog.cc:455] T 00000000000000000000000000000000 P 264c9f6880cb4d77b01bdc78088464dc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "264c9f6880cb4d77b01bdc78088464dc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "264c9f6880cb4d77b01bdc78088464dc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36107 } } }
I20260924 21:10:12.577522 13156 sys_catalog.cc:458] T 00000000000000000000000000000000 P 264c9f6880cb4d77b01bdc78088464dc [sys.catalog]: This master's current role is: LEADER
I20260924 21:10:12.577718 13162 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 21:10:12.578246 13162 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 21:10:12.578526 13150 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790284076800039-392-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260924 21:10:12.578979 13162 catalog_manager.cc:1259] Loaded cluster ID: 6fe3c2ce291b4b30ac8726576398e5e0
I20260924 21:10:12.579070 13162 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 21:10:12.582424 13162 catalog_manager.cc:1504] Loading token signing keys...
I20260924 21:10:12.582640 13162 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 264c9f6880cb4d77b01bdc78088464dc: Loaded TSK: 0
I20260924 21:10:12.582880 13162 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260924 21:10:12.659809 13153 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 21:10:12.660022 13153 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 21:10:12.660059 13153 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 21:10:12.661155 13153 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 21:10:12.661217 13153 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 21:10:12.661267 13153 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 21:10:12.662559 13153 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41027
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790284076800039-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790284076800039-392-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:56145/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790284076800039-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:42427
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790284076800039-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=35689
--tserver_master_addrs=127.0.98.62:36107
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13153
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790284076800039-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 19:49:06 UTC on ccycloud.quasar-vhqjoc.root.comops.site
build id 4670864
I20260924 21:10:12.662770 13153 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 21:10:12.662957 13153 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 21:10:12.671633 13190 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 21:10:12.671985 13188 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 21:10:12.672299 13187 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 21:10:12.672767 13153 server_base.cc:1034] running on GCE node
I20260924 21:10:12.673110 13153 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 21:10:12.673377 13153 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 21:10:12.674525 13153 hybrid_clock.cc:648] HybridClock initialized: now 1790284212674507 us; error 35 us; skew 500 ppm
Sep 24 21:10:12 dist-test-slave-c7tk krb5kdc[12118](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284212, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 21:10:12.677002 13153 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 21:10:12.677531 13153 webserver.cc:460] Webserver started at http://127.0.98.1:35689/ using document root <none> and password file <none>
I20260924 21:10:12.677731 13153 fs_manager.cc:359] Metadata directory not provided
I20260924 21:10:12.677825 13153 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 24 21:10:12 dist-test-slave-c7tk krb5kdc[12118](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284212, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260924 21:10:12.694126 13153 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.001s sys 0.000s
I20260924 21:10:12.702091 13198 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 21:10:12.702250 13153 fs_manager.cc:725] Time spent opening block manager: real 0.008s user 0.001s sys 0.000s
I20260924 21:10:12.702312 13153 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790284076800039-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790284076800039-392-0/minicluster-data/ts-0/wal
uuid: "bc21df88e45348ef92e3b18c1c4dcdcc"
format_stamp: "Formatted at 2026-09-24 21:10:11 on dist-test-slave-c7tk"
server_key: "917e7ca237d4549e22a6882c943375ec"
server_key_iv: "2f1e5ec841d38bdcc08553e12f30fa24"
server_key_version: "kuduclusterkey@0"
I20260924 21:10:12.702656 13153 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790284076800039-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790284076800039-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790284076800039-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 21:10:12.722040 13153 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 21:10:12.724048 13153 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 21:10:12.724225 13153 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 21:10:12.724443 13153 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 21:10:12.724767 13153 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 21:10:12.724815 13153 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 21:10:12.724871 13153 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 21:10:12.724920 13153 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 21:10:12.730680 13153 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:42427
I20260924 21:10:12.731153 13153 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790284076800039-392-0/minicluster-data/ts-0/data/info.pb
I20260924 21:10:12.732770 13311 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:42427 every 8 connection(s)
Sep 24 21:10:12 dist-test-slave-c7tk krb5kdc[12118](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284212, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 21:10:12.741190 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAzNNjS/build/release/bin/kudu as pid 13153
I20260924 21:10:12.741451 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAzNNjS/build/release/bin/kudu
/tmp/dist-test-taskAzNNjS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790284076800039-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790284076800039-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790284076800039-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790284076800039-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:56145/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:34245
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:36107
--webserver_port=41933
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790284076800039-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41027
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:56145/kms with env {KRB5CCNAME=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790284076800039-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790284076800039-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790284076800039-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 21:10:12.743975 13315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:10:12.732601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:59017 (local address 127.0.98.62:36107)
0924 21:10:12.733623 (+ 1022us) server_negotiation.cc:206] Beginning negotiation
0924 21:10:12.733629 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 21:10:12.733652 (+ 23us) server_negotiation.cc:407] Connection header received
0924 21:10:12.733746 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:10:12.733750 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:10:12.733784 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:10:12.733962 (+ 178us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:10:12.734443 (+ 481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:10:12.735467 (+ 1024us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:10:12.736410 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:10:12.736762 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:10:12.736872 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:10:12.739001 (+ 2129us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 21:10:12.739005 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 21:10:12.739005 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 21:10:12.739015 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0924 21:10:12.740655 (+ 1640us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:10:12.742578 (+ 1923us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:10:12.742580 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:10:12.742581 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:10:12.742680 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:10:12.742965 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:10:12.742966 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:10:12.742967 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:10:12.743242 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 21:10:12.743321 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0924 21:10:12.743647 (+ 326us) server_negotiation.cc:299] Negotiation successful
0924 21:10:12.743757 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":978,"thread_start_us":57,"threads_started":1}
I20260924 21:10:12.744196 13314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:10:12.731823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36107 (local address 127.0.98.1:59017)
0924 21:10:12.733046 (+ 1223us) negotiation.cc:107] Waiting for socket to connect
0924 21:10:12.733072 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0924 21:10:12.733485 (+ 413us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 21:10:12.734000 (+ 515us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 21:10:12.734003 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 21:10:12.734168 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0924 21:10:12.734322 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:10:12.734339 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:10:12.735558 (+ 1219us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:10:12.735559 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:10:12.736317 (+ 758us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:10:12.736320 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:10:12.736863 (+ 543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:10:12.736865 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:10:12.736977 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:10:12.737399 (+ 422us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 21:10:12.737406 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0924 21:10:12.738892 (+ 1486us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 21:10:12.740773 (+ 1881us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 21:10:12.740779 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 21:10:12.740781 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 21:10:12.741033 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 21:10:12.742760 (+ 1727us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 21:10:12.742761 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 21:10:12.742762 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 21:10:12.742889 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 21:10:12.743321 (+ 432us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 21:10:12.743328 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 21:10:12.743477 (+ 149us) client_negotiation.cc:769] Sending connection context
0924 21:10:12.743587 (+ 110us) client_negotiation.cc:240] Negotiation successful
0924 21:10:12.743717 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1193,"thread_start_us":503,"threads_started":1}
I20260924 21:10:12.745153 13312 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36107
I20260924 21:10:12.745237 13312 heartbeater.cc:464] Registering TS with master...
I20260924 21:10:12.745452 13312 heartbeater.cc:511] Master 127.0.98.62:36107 requested a full tablet report, sending...
W20260924 21:10:12.746371 13312 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:36107 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260924 21:10:12.746443 13312 heartbeater.cc:464] Registering TS with master...
I20260924 21:10:12.746502 13312 heartbeater.cc:511] Master 127.0.98.62:36107 requested a full tablet report, sending...
I20260924 21:10:12.746850 13312 heartbeater.cc:464] Registering TS with master...
I20260924 21:10:12.746973 13312 heartbeater.cc:511] Master 127.0.98.62:36107 requested a full tablet report, sending...
W20260924 21:10:12.747257 13312 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260924 21:10:12.838306 13316 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 21:10:12.838470 13316 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 21:10:12.838488 13316 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 21:10:12.839586 13316 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 21:10:12.839656 13316 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 21:10:12.839692 13316 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 21:10:12.841598 13316 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41027
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790284076800039-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790284076800039-392-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:56145/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790284076800039-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:34245
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790284076800039-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=41933
--tserver_master_addrs=127.0.98.62:36107
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13316
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790284076800039-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 19:49:06 UTC on ccycloud.quasar-vhqjoc.root.comops.site
build id 4670864
I20260924 21:10:12.841791 13316 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 21:10:12.842039 13316 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 21:10:12.846688 13322 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 21:10:12.846692 13321 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 21:10:12.846973 13324 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 21:10:12.848869 13316 server_base.cc:1034] running on GCE node
I20260924 21:10:12.849093 13316 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 21:10:12.849321 13316 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 21:10:12.850626 13316 hybrid_clock.cc:648] HybridClock initialized: now 1790284212850615 us; error 25 us; skew 500 ppm
Sep 24 21:10:12 dist-test-slave-c7tk krb5kdc[12118](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284212, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 21:10:12.853422 13316 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 21:10:12.853868 13316 webserver.cc:460] Webserver started at http://127.0.98.2:41933/ using document root <none> and password file <none>
I20260924 21:10:12.854102 13316 fs_manager.cc:359] Metadata directory not provided
I20260924 21:10:12.854171 13316 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 24 21:10:12 dist-test-slave-c7tk krb5kdc[12118](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284212, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260924 21:10:12.869582 13316 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 21:10:12.870275 13332 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 21:10:12.870422 13316 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 21:10:12.870503 13316 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790284076800039-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790284076800039-392-0/minicluster-data/ts-1/wal
uuid: "487f4094f26047459a40e54292b6716d"
format_stamp: "Formatted at 2026-09-24 21:10:12 on dist-test-slave-c7tk"
server_key: "c03e3c6975e278fa3be2963a6dc2eaed"
server_key_iv: "3274653c8840e1adcadf1318992e170f"
server_key_version: "kuduclusterkey@0"
I20260924 21:10:12.870824 13316 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790284076800039-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790284076800039-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790284076800039-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 21:10:12.881176 13316 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 21:10:12.883080 13316 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 21:10:12.883195 13316 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 21:10:12.883419 13316 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 21:10:12.883713 13316 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 21:10:12.883766 13316 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 21:10:12.883821 13316 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 21:10:12.883845 13316 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 21:10:12.890303 13316 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:34245
I20260924 21:10:12.890372 13445 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:34245 every 8 connection(s)
I20260924 21:10:12.890755 13316 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790284076800039-392-0/minicluster-data/ts-1/data/info.pb
I20260924 21:10:12.896512 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAzNNjS/build/release/bin/kudu as pid 13316
I20260924 21:10:12.896778 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAzNNjS/build/release/bin/kudu
/tmp/dist-test-taskAzNNjS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790284076800039-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790284076800039-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790284076800039-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790284076800039-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:56145/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:41583
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:36107
--webserver_port=33993
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790284076800039-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41027
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:56145/kms with env {KRB5CCNAME=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790284076800039-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790284076800039-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790284076800039-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 21:10:12 dist-test-slave-c7tk krb5kdc[12118](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284212, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 21:10:12.901978 13315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:10:12.892005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:42971 (local address 127.0.98.62:36107)
0924 21:10:12.892043 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0924 21:10:12.892044 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 21:10:12.894140 (+ 2096us) server_negotiation.cc:407] Connection header received
0924 21:10:12.894595 (+ 455us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:10:12.894597 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:10:12.894610 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:10:12.894656 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:10:12.895125 (+ 469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:10:12.895640 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:10:12.896684 (+ 1044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:10:12.896880 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:10:12.896964 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:10:12.898870 (+ 1906us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 21:10:12.898872 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 21:10:12.898872 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 21:10:12.898880 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 21:10:12.900553 (+ 1673us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:10:12.900996 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:10:12.900997 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:10:12.900998 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:10:12.901056 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:10:12.901274 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:10:12.901275 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:10:12.901276 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:10:12.901409 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 21:10:12.901495 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0924 21:10:12.901786 (+ 291us) server_negotiation.cc:299] Negotiation successful
0924 21:10:12.901886 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260924 21:10:12.902125 13448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:10:12.891408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36107 (local address 127.0.98.2:42971)
0924 21:10:12.894035 (+ 2627us) negotiation.cc:107] Waiting for socket to connect
0924 21:10:12.894072 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0924 21:10:12.894502 (+ 430us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 21:10:12.894713 (+ 211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 21:10:12.894716 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 21:10:12.894876 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0924 21:10:12.895033 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:10:12.895037 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:10:12.895724 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:10:12.895725 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:10:12.896609 (+ 884us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:10:12.896612 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:10:12.896988 (+ 376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:10:12.896989 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:10:12.897088 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:10:12.897514 (+ 426us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 21:10:12.897521 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0924 21:10:12.898778 (+ 1257us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 21:10:12.900663 (+ 1885us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 21:10:12.900668 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 21:10:12.900669 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 21:10:12.900912 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 21:10:12.901124 (+ 212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 21:10:12.901125 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 21:10:12.901125 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 21:10:12.901209 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 21:10:12.901485 (+ 276us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 21:10:12.901490 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 21:10:12.901634 (+ 144us) client_negotiation.cc:769] Sending connection context
0924 21:10:12.901735 (+ 101us) client_negotiation.cc:240] Negotiation successful
0924 21:10:12.901892 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2593,"thread_start_us":2566,"threads_started":1}
I20260924 21:10:12.902595 13446 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36107
I20260924 21:10:12.902689 13446 heartbeater.cc:464] Registering TS with master...
I20260924 21:10:12.902841 13446 heartbeater.cc:511] Master 127.0.98.62:36107 requested a full tablet report, sending...
W20260924 21:10:12.903434 13446 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:36107 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260924 21:10:12.903499 13446 heartbeater.cc:464] Registering TS with master...
I20260924 21:10:12.903553 13446 heartbeater.cc:511] Master 127.0.98.62:36107 requested a full tablet report, sending...
I20260924 21:10:12.903828 13446 heartbeater.cc:464] Registering TS with master...
I20260924 21:10:12.903892 13446 heartbeater.cc:511] Master 127.0.98.62:36107 requested a full tablet report, sending...
W20260924 21:10:12.904163 13446 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260924 21:10:12.990863 13449 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 21:10:12.991081 13449 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 21:10:12.991117 13449 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 21:10:12.992138 13449 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 21:10:12.992198 13449 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 21:10:12.992239 13449 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 21:10:12.993589 13449 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41027
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790284076800039-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790284076800039-392-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:56145/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790284076800039-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:41583
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790284076800039-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=33993
--tserver_master_addrs=127.0.98.62:36107
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13449
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790284076800039-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 19:49:06 UTC on ccycloud.quasar-vhqjoc.root.comops.site
build id 4670864
I20260924 21:10:12.993842 13449 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 21:10:12.994064 13449 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 21:10:12.998660 13457 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 21:10:12.998643 13454 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 21:10:12.999248 13455 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 21:10:12.999718 13449 server_base.cc:1034] running on GCE node
I20260924 21:10:12.999898 13449 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 21:10:13.000126 13449 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 21:10:13.001288 13449 hybrid_clock.cc:648] HybridClock initialized: now 1790284213001274 us; error 26 us; skew 500 ppm
Sep 24 21:10:12 dist-test-slave-c7tk krb5kdc[12118](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284212, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 21:10:13.003465 13449 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 21:10:13.003863 13449 webserver.cc:460] Webserver started at http://127.0.98.3:33993/ using document root <none> and password file <none>
I20260924 21:10:13.004045 13449 fs_manager.cc:359] Metadata directory not provided
I20260924 21:10:13.004115 13449 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 24 21:10:13 dist-test-slave-c7tk krb5kdc[12118](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284212, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260924 21:10:13.027767 13449 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.000s
I20260924 21:10:13.028470 13466 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 21:10:13.028637 13449 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260924 21:10:13.028709 13449 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790284076800039-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790284076800039-392-0/minicluster-data/ts-2/wal
uuid: "bf6df1fafc5149e9861501adbce69453"
format_stamp: "Formatted at 2026-09-24 21:10:12 on dist-test-slave-c7tk"
server_key: "4bc3d96635bdcacf961a9b4cf2bd569c"
server_key_iv: "46e04b1e36f770a8ee4b9d79c3af35e1"
server_key_version: "kuduclusterkey@0"
I20260924 21:10:13.029656 13449 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790284076800039-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790284076800039-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790284076800039-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 21:10:13.044926 13449 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 21:10:13.046958 13449 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 21:10:13.047079 13449 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 21:10:13.047268 13449 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 21:10:13.047577 13449 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 21:10:13.047623 13449 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 21:10:13.047650 13449 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 21:10:13.047674 13449 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 21:10:13.054533 13449 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:41583
I20260924 21:10:13.054591 13579 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:41583 every 8 connection(s)
I20260924 21:10:13.054962 13449 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790284076800039-392-0/minicluster-data/ts-2/data/info.pb
I20260924 21:10:13.056788 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAzNNjS/build/release/bin/kudu as pid 13449
Sep 24 21:10:13 dist-test-slave-c7tk krb5kdc[12118](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284212, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 21:10:13.063831 13315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:10:13.055595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:38781 (local address 127.0.98.62:36107)
0924 21:10:13.055638 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0924 21:10:13.055640 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 21:10:13.055870 (+ 230us) server_negotiation.cc:407] Connection header received
0924 21:10:13.056433 (+ 563us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:10:13.056434 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:10:13.056443 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:10:13.056484 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:10:13.057124 (+ 640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:10:13.057773 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:10:13.058779 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:10:13.059036 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:10:13.059085 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:10:13.061038 (+ 1953us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 21:10:13.061039 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 21:10:13.061039 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 21:10:13.061051 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 21:10:13.062489 (+ 1438us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:10:13.062942 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:10:13.062943 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:10:13.062943 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:10:13.062997 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:10:13.063208 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:10:13.063209 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:10:13.063209 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:10:13.063327 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 21:10:13.063395 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0924 21:10:13.063693 (+ 298us) server_negotiation.cc:299] Negotiation successful
0924 21:10:13.063767 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260924 21:10:13.064200 13582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:10:13.055582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36107 (local address 127.0.98.3:38781)
0924 21:10:13.055743 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0924 21:10:13.055784 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 21:10:13.056328 (+ 544us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 21:10:13.056555 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 21:10:13.056559 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 21:10:13.056774 (+ 215us) client_negotiation.cc:189] Negotiated authn=SASL
0924 21:10:13.057027 (+ 253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:10:13.057034 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:10:13.057851 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:10:13.057852 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:10:13.058694 (+ 842us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:10:13.058696 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:10:13.059152 (+ 456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:10:13.059153 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:10:13.059245 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:10:13.059720 (+ 475us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 21:10:13.059727 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0924 21:10:13.060948 (+ 1221us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 21:10:13.062570 (+ 1622us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 21:10:13.062575 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 21:10:13.062577 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 21:10:13.062853 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 21:10:13.063050 (+ 197us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 21:10:13.063051 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 21:10:13.063051 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 21:10:13.063145 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 21:10:13.063402 (+ 257us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 21:10:13.063414 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 21:10:13.063559 (+ 145us) client_negotiation.cc:769] Sending connection context
0924 21:10:13.063655 (+ 96us) client_negotiation.cc:240] Negotiation successful
0924 21:10:13.063770 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":74,"threads_started":1}
I20260924 21:10:13.064833 13580 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36107
I20260924 21:10:13.064990 13580 heartbeater.cc:464] Registering TS with master...
I20260924 21:10:13.065176 13580 heartbeater.cc:511] Master 127.0.98.62:36107 requested a full tablet report, sending...
I20260924 21:10:13.065598 13584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:10:13.057336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50880 (local address 127.0.98.62:36107)
0924 21:10:13.057449 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0924 21:10:13.057451 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 21:10:13.057597 (+ 146us) server_negotiation.cc:407] Connection header received
0924 21:10:13.057654 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:10:13.057655 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:10:13.057664 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:10:13.057728 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:10:13.058409 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:10:13.059063 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:10:13.060766 (+ 1703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:10:13.061056 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:10:13.061115 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:10:13.062615 (+ 1500us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 21:10:13.062616 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 21:10:13.062616 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 21:10:13.062622 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 21:10:13.064432 (+ 1810us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:10:13.064804 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:10:13.064805 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:10:13.064805 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:10:13.064863 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:10:13.065098 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:10:13.065099 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:10:13.065099 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:10:13.065214 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 21:10:13.065361 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0924 21:10:13.065441 (+ 80us) server_negotiation.cc:299] Negotiation successful
0924 21:10:13.065518 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95,"thread_start_us":57,"threads_started":1}
W20260924 21:10:13.065714 13580 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:36107 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260924 21:10:13.065776 13580 heartbeater.cc:464] Registering TS with master...
I20260924 21:10:13.065855 13580 heartbeater.cc:511] Master 127.0.98.62:36107 requested a full tablet report, sending...
I20260924 21:10:13.066231 13580 heartbeater.cc:464] Registering TS with master...
I20260924 21:10:13.066303 13580 heartbeater.cc:511] Master 127.0.98.62:36107 requested a full tablet report, sending...
W20260924 21:10:13.066557 13580 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 24 21:10:13 dist-test-slave-c7tk krb5kdc[12118](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790284213, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 21:10:13.747435 13312 heartbeater.cc:464] Registering TS with master...
I20260924 21:10:13.747581 13312 heartbeater.cc:511] Master 127.0.98.62:36107 requested a full tablet report, sending...
I20260924 21:10:13.747887 13113 ts_manager.cc:194] Registered new tserver with Master: bc21df88e45348ef92e3b18c1c4dcdcc (127.0.98.1:42427)
I20260924 21:10:13.748484 13113 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:59017
Sep 24 21:10:13 dist-test-slave-c7tk krb5kdc[12118](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284213, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260924 21:10:13.904390 13446 heartbeater.cc:464] Registering TS with master...
I20260924 21:10:13.904520 13446 heartbeater.cc:511] Master 127.0.98.62:36107 requested a full tablet report, sending...
I20260924 21:10:13.904843 13113 ts_manager.cc:194] Registered new tserver with Master: 487f4094f26047459a40e54292b6716d (127.0.98.2:34245)
I20260924 21:10:13.905395 13113 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:42971
I20260924 21:10:14.068382 13580 heartbeater.cc:464] Registering TS with master...
I20260924 21:10:14.068501 13580 heartbeater.cc:511] Master 127.0.98.62:36107 requested a full tablet report, sending...
I20260924 21:10:14.068804 13113 ts_manager.cc:194] Registered new tserver with Master: bf6df1fafc5149e9861501adbce69453 (127.0.98.3:41583)
I20260924 21:10:14.069247 13113 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:38781
I20260924 21:10:14.070549 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 24 21:10:14 dist-test-slave-c7tk krb5kdc[12118](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284213, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260924 21:10:14.091660 13603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:10:14.072555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50896 (local address 127.0.98.62:36107)
0924 21:10:14.075434 (+ 2879us) server_negotiation.cc:206] Beginning negotiation
0924 21:10:14.075436 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 21:10:14.075552 (+ 116us) server_negotiation.cc:407] Connection header received
0924 21:10:14.075641 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:10:14.075642 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:10:14.075652 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:10:14.075689 (+ 37us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:10:14.076326 (+ 637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:10:14.076930 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:10:14.077756 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:10:14.078026 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:10:14.078069 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:10:14.082628 (+ 4559us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 21:10:14.082629 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 21:10:14.082629 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 21:10:14.082636 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 21:10:14.084175 (+ 1539us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:10:14.088996 (+ 4821us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:10:14.088997 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:10:14.088997 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:10:14.089055 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:10:14.091052 (+ 1997us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:10:14.091053 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:10:14.091054 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:10:14.091182 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 21:10:14.091263 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0924 21:10:14.091467 (+ 204us) server_negotiation.cc:299] Negotiation successful
0924 21:10:14.091543 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2858,"thread_start_us":48,"threads_started":1}
I20260924 21:10:14.093703 13113 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50896:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 21:10:14.095974 13113 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 21:10:14.105080 13609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:10:14.099726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:41583 (local address 127.0.0.1:53730)
0924 21:10:14.100726 (+ 1000us) negotiation.cc:107] Waiting for socket to connect
0924 21:10:14.100765 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0924 21:10:14.100836 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 21:10:14.101215 (+ 379us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 21:10:14.101218 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 21:10:14.101220 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 21:10:14.101275 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:10:14.101277 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:10:14.102666 (+ 1389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:10:14.102667 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:10:14.103772 (+ 1105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:10:14.103774 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:10:14.104824 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:10:14.104824 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:10:14.104894 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:10:14.104917 (+ 23us) client_negotiation.cc:769] Sending connection context
0924 21:10:14.104966 (+ 49us) client_negotiation.cc:240] Negotiation successful
0924 21:10:14.105013 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":971,"thread_start_us":87,"threads_started":1}
I20260924 21:10:14.105533 13610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:10:14.099971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53730 (local address 127.0.98.3:41583)
0924 21:10:14.100994 (+ 1023us) server_negotiation.cc:206] Beginning negotiation
0924 21:10:14.100999 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 21:10:14.101008 (+ 9us) server_negotiation.cc:407] Connection header received
0924 21:10:14.101038 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:10:14.101040 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:10:14.101054 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:10:14.101121 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 21:10:14.101348 (+ 227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:10:14.102586 (+ 1238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:10:14.103861 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:10:14.104502 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:10:14.105321 (+ 819us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:10:14.105381 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0924 21:10:14.105402 (+ 21us) server_negotiation.cc:299] Negotiation successful
0924 21:10:14.105470 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":992,"thread_start_us":49,"threads_started":1}
I20260924 21:10:14.106196 13514 tablet_service.cc:1467] Processing CreateTablet for tablet caf1042e243443d78b15e7c39b8b9235 (DEFAULT_TABLE table=test-table [id=be9f415047fe47a4a6b5bbb4996bc94f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 21:10:14.106530 13514 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet caf1042e243443d78b15e7c39b8b9235. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 21:10:14.109933 13612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:10:14.100247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34245 (local address 127.0.0.1:48940)
0924 21:10:14.106179 (+ 5932us) negotiation.cc:107] Waiting for socket to connect
0924 21:10:14.106189 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 21:10:14.106229 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 21:10:14.106540 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 21:10:14.106541 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 21:10:14.106543 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 21:10:14.106575 (+ 32us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:10:14.106577 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:10:14.107833 (+ 1256us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:10:14.107834 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:10:14.108711 (+ 877us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:10:14.108713 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:10:14.109662 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:10:14.109664 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:10:14.109727 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:10:14.109731 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 21:10:14.109779 (+ 48us) client_negotiation.cc:240] Negotiation successful
0924 21:10:14.109820 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5916,"thread_start_us":79,"threads_started":1}
I20260924 21:10:14.110239 13613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:10:14.100447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48940 (local address 127.0.98.2:34245)
0924 21:10:14.106325 (+ 5878us) server_negotiation.cc:206] Beginning negotiation
0924 21:10:14.106330 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 21:10:14.106339 (+ 9us) server_negotiation.cc:407] Connection header received
0924 21:10:14.106369 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:10:14.106371 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:10:14.106384 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:10:14.106450 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 21:10:14.106673 (+ 223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:10:14.107755 (+ 1082us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:10:14.108790 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:10:14.109465 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:10:14.109535 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:10:14.109608 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0924 21:10:14.110113 (+ 505us) server_negotiation.cc:299] Negotiation successful
0924 21:10:14.110152 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5847,"thread_start_us":56,"threads_started":1}
I20260924 21:10:14.110553 13614 tablet_bootstrap.cc:492] T caf1042e243443d78b15e7c39b8b9235 P bf6df1fafc5149e9861501adbce69453: Bootstrap starting.
I20260924 21:10:14.110985 13371 tablet_service.cc:1467] Processing CreateTablet for tablet caf1042e243443d78b15e7c39b8b9235 (DEFAULT_TABLE table=test-table [id=be9f415047fe47a4a6b5bbb4996bc94f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 21:10:14.111248 13371 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet caf1042e243443d78b15e7c39b8b9235. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 21:10:14.111436 13614 tablet_bootstrap.cc:654] T caf1042e243443d78b15e7c39b8b9235 P bf6df1fafc5149e9861501adbce69453: Neither blocks nor log segments found. Creating new log.
I20260924 21:10:14.111804 13614 log.cc:824] T caf1042e243443d78b15e7c39b8b9235 P bf6df1fafc5149e9861501adbce69453: Log is configured to *not* fsync() on all Append() calls
I20260924 21:10:14.113200 13614 tablet_bootstrap.cc:492] T caf1042e243443d78b15e7c39b8b9235 P bf6df1fafc5149e9861501adbce69453: No bootstrap required, opened a new log
I20260924 21:10:14.113250 13614 ts_tablet_manager.cc:1397] T caf1042e243443d78b15e7c39b8b9235 P bf6df1fafc5149e9861501adbce69453: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260924 21:10:14.113989 13609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:10:14.099726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42427 (local address 127.0.0.1:40222)
0924 21:10:14.105219 (+ 5493us) negotiation.cc:107] Waiting for socket to connect
0924 21:10:14.105226 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0924 21:10:14.105255 (+ 29us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 21:10:14.110125 (+ 4870us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 21:10:14.110126 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 21:10:14.110127 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 21:10:14.110154 (+ 27us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:10:14.110155 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:10:14.111601 (+ 1446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:10:14.111603 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:10:14.112675 (+ 1072us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:10:14.112678 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:10:14.113758 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:10:14.113759 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:10:14.113849 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:10:14.113853 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 21:10:14.113883 (+ 30us) client_negotiation.cc:240] Negotiation successful
0924 21:10:14.113922 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5483,"mutex_wait_us":2,"thread_start_us":66,"threads_started":1}
I20260924 21:10:14.114343 13611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:10:14.100168 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40222 (local address 127.0.98.1:42427)
0924 21:10:14.109953 (+ 9785us) server_negotiation.cc:206] Beginning negotiation
0924 21:10:14.109957 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 21:10:14.109969 (+ 12us) server_negotiation.cc:407] Connection header received
0924 21:10:14.110006 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:10:14.110008 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:10:14.110025 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:10:14.110221 (+ 196us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 21:10:14.110299 (+ 78us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:10:14.111488 (+ 1189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:10:14.112937 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:10:14.113650 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:10:14.114139 (+ 489us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:10:14.114217 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0924 21:10:14.114241 (+ 24us) server_negotiation.cc:299] Negotiation successful
0924 21:10:14.114276 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9752,"thread_start_us":55,"threads_started":1}
I20260924 21:10:14.115015 13227 tablet_service.cc:1467] Processing CreateTablet for tablet caf1042e243443d78b15e7c39b8b9235 (DEFAULT_TABLE table=test-table [id=be9f415047fe47a4a6b5bbb4996bc94f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 21:10:14.115280 13227 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet caf1042e243443d78b15e7c39b8b9235. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 21:10:14.115576 13614 raft_consensus.cc:348] T caf1042e243443d78b15e7c39b8b9235 P bf6df1fafc5149e9861501adbce69453 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf6df1fafc5149e9861501adbce69453" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41583 } } peers { permanent_uuid: "bc21df88e45348ef92e3b18c1c4dcdcc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42427 } } peers { permanent_uuid: "487f4094f26047459a40e54292b6716d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34245 } }
I20260924 21:10:14.115700 13614 raft_consensus.cc:374] T caf1042e243443d78b15e7c39b8b9235 P bf6df1fafc5149e9861501adbce69453 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 21:10:14.115727 13614 raft_consensus.cc:729] T caf1042e243443d78b15e7c39b8b9235 P bf6df1fafc5149e9861501adbce69453 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf6df1fafc5149e9861501adbce69453, State: Initialized, Role: FOLLOWER
I20260924 21:10:14.115816 13614 consensus_queue.cc:260] T caf1042e243443d78b15e7c39b8b9235 P bf6df1fafc5149e9861501adbce69453 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf6df1fafc5149e9861501adbce69453" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41583 } } peers { permanent_uuid: "bc21df88e45348ef92e3b18c1c4dcdcc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42427 } } peers { permanent_uuid: "487f4094f26047459a40e54292b6716d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34245 } }
I20260924 21:10:14.116029 13614 ts_tablet_manager.cc:1428] T caf1042e243443d78b15e7c39b8b9235 P bf6df1fafc5149e9861501adbce69453: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260924 21:10:14.116216 13580 heartbeater.cc:503] Master 127.0.98.62:36107 was elected leader, sending a full tablet report...
I20260924 21:10:14.118081 13617 tablet_bootstrap.cc:492] T caf1042e243443d78b15e7c39b8b9235 P 487f4094f26047459a40e54292b6716d: Bootstrap starting.
I20260924 21:10:14.118726 13617 tablet_bootstrap.cc:654] T caf1042e243443d78b15e7c39b8b9235 P 487f4094f26047459a40e54292b6716d: Neither blocks nor log segments found. Creating new log.
I20260924 21:10:14.119065 13617 log.cc:824] T caf1042e243443d78b15e7c39b8b9235 P 487f4094f26047459a40e54292b6716d: Log is configured to *not* fsync() on all Append() calls
I20260924 21:10:14.120760 13617 tablet_bootstrap.cc:492] T caf1042e243443d78b15e7c39b8b9235 P 487f4094f26047459a40e54292b6716d: No bootstrap required, opened a new log
I20260924 21:10:14.120823 13617 ts_tablet_manager.cc:1397] T caf1042e243443d78b15e7c39b8b9235 P 487f4094f26047459a40e54292b6716d: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260924 21:10:14.123317 13617 raft_consensus.cc:348] T caf1042e243443d78b15e7c39b8b9235 P 487f4094f26047459a40e54292b6716d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf6df1fafc5149e9861501adbce69453" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41583 } } peers { permanent_uuid: "bc21df88e45348ef92e3b18c1c4dcdcc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42427 } } peers { permanent_uuid: "487f4094f26047459a40e54292b6716d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34245 } }
I20260924 21:10:14.123461 13617 raft_consensus.cc:374] T caf1042e243443d78b15e7c39b8b9235 P 487f4094f26047459a40e54292b6716d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 21:10:14.123505 13617 raft_consensus.cc:729] T caf1042e243443d78b15e7c39b8b9235 P 487f4094f26047459a40e54292b6716d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 487f4094f26047459a40e54292b6716d, State: Initialized, Role: FOLLOWER
I20260924 21:10:14.123636 13617 consensus_queue.cc:260] T caf1042e243443d78b15e7c39b8b9235 P 487f4094f26047459a40e54292b6716d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf6df1fafc5149e9861501adbce69453" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41583 } } peers { permanent_uuid: "bc21df88e45348ef92e3b18c1c4dcdcc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42427 } } peers { permanent_uuid: "487f4094f26047459a40e54292b6716d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34245 } }
I20260924 21:10:14.123884 13617 ts_tablet_manager.cc:1428] T caf1042e243443d78b15e7c39b8b9235 P 487f4094f26047459a40e54292b6716d: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 21:10:14.123996 13446 heartbeater.cc:503] Master 127.0.98.62:36107 was elected leader, sending a full tablet report...
I20260924 21:10:14.124231 13620 tablet_bootstrap.cc:492] T caf1042e243443d78b15e7c39b8b9235 P bc21df88e45348ef92e3b18c1c4dcdcc: Bootstrap starting.
I20260924 21:10:14.124842 13620 tablet_bootstrap.cc:654] T caf1042e243443d78b15e7c39b8b9235 P bc21df88e45348ef92e3b18c1c4dcdcc: Neither blocks nor log segments found. Creating new log.
I20260924 21:10:14.125187 13620 log.cc:824] T caf1042e243443d78b15e7c39b8b9235 P bc21df88e45348ef92e3b18c1c4dcdcc: Log is configured to *not* fsync() on all Append() calls
I20260924 21:10:14.126116 13620 tablet_bootstrap.cc:492] T caf1042e243443d78b15e7c39b8b9235 P bc21df88e45348ef92e3b18c1c4dcdcc: No bootstrap required, opened a new log
I20260924 21:10:14.126197 13620 ts_tablet_manager.cc:1397] T caf1042e243443d78b15e7c39b8b9235 P bc21df88e45348ef92e3b18c1c4dcdcc: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 21:10:14.128329 13620 raft_consensus.cc:348] T caf1042e243443d78b15e7c39b8b9235 P bc21df88e45348ef92e3b18c1c4dcdcc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf6df1fafc5149e9861501adbce69453" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41583 } } peers { permanent_uuid: "bc21df88e45348ef92e3b18c1c4dcdcc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42427 } } peers { permanent_uuid: "487f4094f26047459a40e54292b6716d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34245 } }
I20260924 21:10:14.128510 13620 raft_consensus.cc:374] T caf1042e243443d78b15e7c39b8b9235 P bc21df88e45348ef92e3b18c1c4dcdcc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 21:10:14.128571 13620 raft_consensus.cc:729] T caf1042e243443d78b15e7c39b8b9235 P bc21df88e45348ef92e3b18c1c4dcdcc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc21df88e45348ef92e3b18c1c4dcdcc, State: Initialized, Role: FOLLOWER
I20260924 21:10:14.128706 13620 consensus_queue.cc:260] T caf1042e243443d78b15e7c39b8b9235 P bc21df88e45348ef92e3b18c1c4dcdcc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf6df1fafc5149e9861501adbce69453" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41583 } } peers { permanent_uuid: "bc21df88e45348ef92e3b18c1c4dcdcc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42427 } } peers { permanent_uuid: "487f4094f26047459a40e54292b6716d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34245 } }
I20260924 21:10:14.129014 13620 ts_tablet_manager.cc:1428] T caf1042e243443d78b15e7c39b8b9235 P bc21df88e45348ef92e3b18c1c4dcdcc: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260924 21:10:14.129130 13312 heartbeater.cc:503] Master 127.0.98.62:36107 was elected leader, sending a full tablet report...
W20260924 21:10:14.145925 13447 tablet.cc:2369] T caf1042e243443d78b15e7c39b8b9235 P 487f4094f26047459a40e54292b6716d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 21:10:14.233877 13313 tablet.cc:2369] T caf1042e243443d78b15e7c39b8b9235 P bc21df88e45348ef92e3b18c1c4dcdcc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 21:10:14.235108 13618 raft_consensus.cc:482] T caf1042e243443d78b15e7c39b8b9235 P bf6df1fafc5149e9861501adbce69453 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 21:10:14.235216 13618 raft_consensus.cc:504] T caf1042e243443d78b15e7c39b8b9235 P bf6df1fafc5149e9861501adbce69453 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf6df1fafc5149e9861501adbce69453" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41583 } } peers { permanent_uuid: "bc21df88e45348ef92e3b18c1c4dcdcc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42427 } } peers { permanent_uuid: "487f4094f26047459a40e54292b6716d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34245 } }
I20260924 21:10:14.235536 13618 leader_election.cc:288] T caf1042e243443d78b15e7c39b8b9235 P bf6df1fafc5149e9861501adbce69453 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bc21df88e45348ef92e3b18c1c4dcdcc (127.0.98.1:42427), 487f4094f26047459a40e54292b6716d (127.0.98.2:34245)
I20260924 21:10:14.247454 13611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:10:14.236152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:39611 (local address 127.0.98.1:42427)
0924 21:10:14.236230 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0924 21:10:14.236232 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 21:10:14.238811 (+ 2579us) server_negotiation.cc:407] Connection header received
0924 21:10:14.238917 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:10:14.238918 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:10:14.238928 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:10:14.238965 (+ 37us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 21:10:14.239205 (+ 240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:10:14.239904 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:10:14.245836 (+ 5932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:10:14.246345 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:10:14.246390 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:10:14.246399 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0924 21:10:14.247357 (+ 958us) server_negotiation.cc:299] Negotiation successful
0924 21:10:14.247394 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260924 21:10:14.247821 13625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:10:14.235740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42427 (local address 127.0.98.3:39611)
0924 21:10:14.238745 (+ 3005us) negotiation.cc:107] Waiting for socket to connect
0924 21:10:14.238757 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 21:10:14.238838 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 21:10:14.239063 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 21:10:14.239063 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 21:10:14.239065 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 21:10:14.239148 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:10:14.239150 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:10:14.240023 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:10:14.240024 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:10:14.241181 (+ 1157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:10:14.241183 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:10:14.246468 (+ 5285us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:10:14.246469 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:10:14.246537 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:10:14.246540 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 21:10:14.247697 (+ 1157us) client_negotiation.cc:240] Negotiation successful
0924 21:10:14.247735 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2987,"thread_start_us":73,"threads_started":1}
I20260924 21:10:14.248126 13613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:10:14.236193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:49017 (local address 127.0.98.2:34245)
0924 21:10:14.236257 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0924 21:10:14.236259 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 21:10:14.243792 (+ 7533us) server_negotiation.cc:407] Connection header received
0924 21:10:14.243896 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:10:14.243897 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:10:14.243907 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:10:14.243933 (+ 26us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 21:10:14.244124 (+ 191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:10:14.244686 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:10:14.246612 (+ 1926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:10:14.247121 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:10:14.247162 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:10:14.247170 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0924 21:10:14.248032 (+ 862us) server_negotiation.cc:299] Negotiation successful
0924 21:10:14.248070 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260924 21:10:14.248413 13624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:10:14.235915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34245 (local address 127.0.98.3:49017)
0924 21:10:14.243740 (+ 7825us) negotiation.cc:107] Waiting for socket to connect
0924 21:10:14.243749 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 21:10:14.243816 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 21:10:14.243990 (+ 174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 21:10:14.243991 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 21:10:14.243992 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 21:10:14.244059 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:10:14.244060 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:10:14.244771 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:10:14.244772 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:10:14.245717 (+ 945us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:10:14.245720 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:10:14.247221 (+ 1501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:10:14.247221 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:10:14.247287 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:10:14.247289 (+ 2us) client_negotiation.cc:769] Sending connection context
0924 21:10:14.248317 (+ 1028us) client_negotiation.cc:240] Negotiation successful
0924 21:10:14.248351 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7808,"thread_start_us":51,"threads_started":1}
I20260924 21:10:14.249557 13266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "caf1042e243443d78b15e7c39b8b9235" candidate_uuid: "bf6df1fafc5149e9861501adbce69453" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc21df88e45348ef92e3b18c1c4dcdcc" is_pre_election: true
I20260924 21:10:14.249735 13266 raft_consensus.cc:2454] T caf1042e243443d78b15e7c39b8b9235 P bc21df88e45348ef92e3b18c1c4dcdcc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bf6df1fafc5149e9861501adbce69453 in term 0.
I20260924 21:10:14.249874 13400 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "caf1042e243443d78b15e7c39b8b9235" candidate_uuid: "bf6df1fafc5149e9861501adbce69453" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "487f4094f26047459a40e54292b6716d" is_pre_election: true
I20260924 21:10:14.249971 13468 leader_election.cc:302] T caf1042e243443d78b15e7c39b8b9235 P bf6df1fafc5149e9861501adbce69453 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bc21df88e45348ef92e3b18c1c4dcdcc, bf6df1fafc5149e9861501adbce69453; no voters:
I20260924 21:10:14.250029 13400 raft_consensus.cc:2454] T caf1042e243443d78b15e7c39b8b9235 P 487f4094f26047459a40e54292b6716d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bf6df1fafc5149e9861501adbce69453 in term 0.
I20260924 21:10:14.250192 13618 raft_consensus.cc:2781] T caf1042e243443d78b15e7c39b8b9235 P bf6df1fafc5149e9861501adbce69453 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 21:10:14.250248 13618 raft_consensus.cc:482] T caf1042e243443d78b15e7c39b8b9235 P bf6df1fafc5149e9861501adbce69453 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 21:10:14.250270 13618 raft_consensus.cc:3037] T caf1042e243443d78b15e7c39b8b9235 P bf6df1fafc5149e9861501adbce69453 [term 0 FOLLOWER]: Advancing to term 1
I20260924 21:10:14.251151 13618 raft_consensus.cc:504] T caf1042e243443d78b15e7c39b8b9235 P bf6df1fafc5149e9861501adbce69453 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf6df1fafc5149e9861501adbce69453" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41583 } } peers { permanent_uuid: "bc21df88e45348ef92e3b18c1c4dcdcc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42427 } } peers { permanent_uuid: "487f4094f26047459a40e54292b6716d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34245 } }
I20260924 21:10:14.251261 13618 leader_election.cc:288] T caf1042e243443d78b15e7c39b8b9235 P bf6df1fafc5149e9861501adbce69453 [CANDIDATE]: Term 1 election: Requested vote from peers bc21df88e45348ef92e3b18c1c4dcdcc (127.0.98.1:42427), 487f4094f26047459a40e54292b6716d (127.0.98.2:34245)
I20260924 21:10:14.251668 13400 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "caf1042e243443d78b15e7c39b8b9235" candidate_uuid: "bf6df1fafc5149e9861501adbce69453" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "487f4094f26047459a40e54292b6716d"
I20260924 21:10:14.251735 13400 raft_consensus.cc:3037] T caf1042e243443d78b15e7c39b8b9235 P 487f4094f26047459a40e54292b6716d [term 0 FOLLOWER]: Advancing to term 1
I20260924 21:10:14.252521 13400 raft_consensus.cc:2454] T caf1042e243443d78b15e7c39b8b9235 P 487f4094f26047459a40e54292b6716d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bf6df1fafc5149e9861501adbce69453 in term 1.
I20260924 21:10:14.252674 13467 leader_election.cc:302] T caf1042e243443d78b15e7c39b8b9235 P bf6df1fafc5149e9861501adbce69453 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 487f4094f26047459a40e54292b6716d, bf6df1fafc5149e9861501adbce69453; no voters:
I20260924 21:10:14.252826 13266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "caf1042e243443d78b15e7c39b8b9235" candidate_uuid: "bf6df1fafc5149e9861501adbce69453" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc21df88e45348ef92e3b18c1c4dcdcc"
I20260924 21:10:14.252903 13266 raft_consensus.cc:3037] T caf1042e243443d78b15e7c39b8b9235 P bc21df88e45348ef92e3b18c1c4dcdcc [term 0 FOLLOWER]: Advancing to term 1
I20260924 21:10:14.253674 13266 raft_consensus.cc:2454] T caf1042e243443d78b15e7c39b8b9235 P bc21df88e45348ef92e3b18c1c4dcdcc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bf6df1fafc5149e9861501adbce69453 in term 1.
I20260924 21:10:14.253929 13618 raft_consensus.cc:2781] T caf1042e243443d78b15e7c39b8b9235 P bf6df1fafc5149e9861501adbce69453 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 21:10:14.254065 13618 raft_consensus.cc:686] T caf1042e243443d78b15e7c39b8b9235 P bf6df1fafc5149e9861501adbce69453 [term 1 LEADER]: Becoming Leader. State: Replica: bf6df1fafc5149e9861501adbce69453, State: Running, Role: LEADER
I20260924 21:10:14.254141 13618 consensus_queue.cc:237] T caf1042e243443d78b15e7c39b8b9235 P bf6df1fafc5149e9861501adbce69453 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf6df1fafc5149e9861501adbce69453" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41583 } } peers { permanent_uuid: "bc21df88e45348ef92e3b18c1c4dcdcc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42427 } } peers { permanent_uuid: "487f4094f26047459a40e54292b6716d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34245 } }
I20260924 21:10:14.254912 13113 catalog_manager.cc:5495] T caf1042e243443d78b15e7c39b8b9235 P bf6df1fafc5149e9861501adbce69453 reported cstate change: term changed from 0 to 1, leader changed from <none> to bf6df1fafc5149e9861501adbce69453 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "bf6df1fafc5149e9861501adbce69453" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf6df1fafc5149e9861501adbce69453" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41583 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bc21df88e45348ef92e3b18c1c4dcdcc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42427 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "487f4094f26047459a40e54292b6716d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34245 } health_report { overall_health: UNKNOWN } } }
I20260924 21:10:14.284605 13610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:10:14.281593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53746 (local address 127.0.98.3:41583)
0924 21:10:14.281696 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0924 21:10:14.281698 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 21:10:14.281705 (+ 7us) server_negotiation.cc:407] Connection header received
0924 21:10:14.281737 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:10:14.281738 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:10:14.281757 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:10:14.281842 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 21:10:14.282039 (+ 197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:10:14.282650 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:10:14.283645 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:10:14.283894 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:10:14.283965 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:10:14.284146 (+ 181us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 21:10:14.284251 (+ 105us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 21:10:14.284319 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0924 21:10:14.284515 (+ 196us) server_negotiation.cc:299] Negotiation successful
0924 21:10:14.284534 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260924 21:10:14.286541 13400 raft_consensus.cc:1261] T caf1042e243443d78b15e7c39b8b9235 P 487f4094f26047459a40e54292b6716d [term 1 FOLLOWER]: Refusing update from remote peer bf6df1fafc5149e9861501adbce69453: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 21:10:14.286928 13626 consensus_queue.cc:1035] T caf1042e243443d78b15e7c39b8b9235 P bf6df1fafc5149e9861501adbce69453 [LEADER]: Connected to new peer: Peer: permanent_uuid: "487f4094f26047459a40e54292b6716d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34245 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 21:10:14.288795 13266 raft_consensus.cc:1261] T caf1042e243443d78b15e7c39b8b9235 P bc21df88e45348ef92e3b18c1c4dcdcc [term 1 FOLLOWER]: Refusing update from remote peer bf6df1fafc5149e9861501adbce69453: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 21:10:14.289093 13626 consensus_queue.cc:1035] T caf1042e243443d78b15e7c39b8b9235 P bf6df1fafc5149e9861501adbce69453 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bc21df88e45348ef92e3b18c1c4dcdcc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42427 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 21:10:14.292466 13629 mvcc.cc:205] Tried to move back new op lower bound from 7333004141715091456 to 7333004141585408000. Current Snapshot: MvccSnapshot[applied={T|T < 7333004141715091456}]
I20260924 21:10:14.292920 13632 mvcc.cc:205] Tried to move back new op lower bound from 7333004141715091456 to 7333004141585408000. Current Snapshot: MvccSnapshot[applied={T|T < 7333004141715091456}]
I20260924 21:10:14.294875 13630 mvcc.cc:205] Tried to move back new op lower bound from 7333004141715091456 to 7333004141585408000. Current Snapshot: MvccSnapshot[applied={T|T < 7333004141715091456}]
I20260924 21:10:14.296759 13113 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50896:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 21:10:14.296836 13113 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50896:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 21:10:14.297513 13113 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 264c9f6880cb4d77b01bdc78088464dc: Sending DeleteTablet for 3 replicas of tablet caf1042e243443d78b15e7c39b8b9235
I20260924 21:10:14.297923 13514 tablet_service.cc:1514] Processing DeleteTablet for tablet caf1042e243443d78b15e7c39b8b9235 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 21:10:14 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:53730
I20260924 21:10:14.297925 13371 tablet_service.cc:1514] Processing DeleteTablet for tablet caf1042e243443d78b15e7c39b8b9235 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 21:10:14 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:48940
I20260924 21:10:14.298247 13227 tablet_service.cc:1514] Processing DeleteTablet for tablet caf1042e243443d78b15e7c39b8b9235 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 21:10:14 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:40222
I20260924 21:10:14.298753 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAzNNjS/build/release/bin/kudu with pid 13153
I20260924 21:10:14.302497 13639 tablet_replica.cc:330] T caf1042e243443d78b15e7c39b8b9235 P bf6df1fafc5149e9861501adbce69453: stopping tablet replica
I20260924 21:10:14.302623 13639 raft_consensus.cc:2229] T caf1042e243443d78b15e7c39b8b9235 P bf6df1fafc5149e9861501adbce69453 [term 1 LEADER]: Raft consensus shutting down.
I20260924 21:10:14.302781 13639 raft_consensus.cc:2258] T caf1042e243443d78b15e7c39b8b9235 P bf6df1fafc5149e9861501adbce69453 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260924 21:10:14.305753 13099 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:42427 (error 108)
I20260924 21:10:14.307027 13639 ts_tablet_manager.cc:1905] T caf1042e243443d78b15e7c39b8b9235 P bf6df1fafc5149e9861501adbce69453: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 21:10:14.308475 13639 ts_tablet_manager.cc:1918] T caf1042e243443d78b15e7c39b8b9235 P bf6df1fafc5149e9861501adbce69453: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 21:10:14.308689 13639 log.cc:1196] T caf1042e243443d78b15e7c39b8b9235 P bf6df1fafc5149e9861501adbce69453: Deleting WAL directory at /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790284076800039-392-0/minicluster-data/ts-2/wal/wals/caf1042e243443d78b15e7c39b8b9235
I20260924 21:10:14.309031 13639 ts_tablet_manager.cc:1939] T caf1042e243443d78b15e7c39b8b9235 P bf6df1fafc5149e9861501adbce69453: Deleting consensus metadata
I20260924 21:10:14.309445 13101 catalog_manager.cc:4841] TS bf6df1fafc5149e9861501adbce69453 (127.0.98.3:41583): tablet caf1042e243443d78b15e7c39b8b9235 (table test-table [id=be9f415047fe47a4a6b5bbb4996bc94f]) successfully deleted
I20260924 21:10:14.310213 13609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:10:14.310106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42427 (local address 127.0.0.1:40228)
0924 21:10:14.310151 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0924 21:10:14.310182 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:42427: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260924 21:10:14.310392 13099 catalog_manager.cc:4584] TS bc21df88e45348ef92e3b18c1c4dcdcc (127.0.98.1:42427): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet caf1042e243443d78b15e7c39b8b9235: Network error: Client connection negotiation failed: client connection to 127.0.98.1:42427: connect: Connection refused (error 111)
I20260924 21:10:14.310603 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAzNNjS/build/release/bin/kudu with pid 13316
W20260924 21:10:14.317759 13609 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 21:10:14.317406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34245 (local address 127.0.0.1:48942)
0924 21:10:14.317457 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0924 21:10:14.317466 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 21:10:14.317500 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 21:10:14.317709 (+ 209us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:34245: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":36}
I20260924 21:10:14.318342 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAzNNjS/build/release/bin/kudu with pid 13449
I20260924 21:10:14.331449 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAzNNjS/build/release/bin/kudu with pid 13081
I20260924 21:10:14.337728 392 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260924 21:10:14.682027 392 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260924 21:10:14.682126 392 mini_ranger.cc:66] Stopping Ranger...
I20260924 21:10:15.026634 392 mini_ranger.cc:68] Stopped Ranger
2026-09-24 21:10:15.026 UTC [12148] LOG: received smart shutdown request
2026-09-24 21:10:15.030 UTC [12148] LOG: background worker "logical replication launcher" (PID 12157) exited with exit code 1
2026-09-24 21:10:15.030 UTC [12152] LOG: shutting down
2026-09-24 21:10:15.178 UTC [12148] LOG: database system is shut down
2026-09-24T21:10:15Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (41965 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790284076800039-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 21:10:15 dist-test-slave-c7tk krb5kdc[13654](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 21:10:15 dist-test-slave-c7tk krb5kdc[13654](info): set up 2 sockets
Sep 24 21:10:15 dist-test-slave-c7tk krb5kdc[13654](info): commencing operation
krb5kdc: starting...
W20260924 21:10:16.205080 392 mini_kdc.cc:121] Time spent starting KDC: real 0.906s user 0.004s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790284076800039-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790284076800039-392-0/krb5kdc/test-user.keytab.
Sep 24 21:10:16 dist-test-slave-c7tk krb5kdc[13654](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284216, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T21:10:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T21:10:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790284076800039-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790284076800039-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 21:10:16.305085 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAzNNjS/build/release/bin/kudu
/tmp/dist-test-taskAzNNjS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790284076800039-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790284076800039-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790284076800039-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790284076800039-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46431
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790284076800039-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39903
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46431,127.0.98.61:39919,127.0.98.60:44849
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790284076800039-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790284076800039-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790284076800039-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 21:10:16.376644 13667 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 21:10:16.376827 13667 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 21:10:16.376858 13667 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 21:10:16.377816 13667 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 21:10:16.377869 13667 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 21:10:16.377889 13667 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 21:10:16.377909 13667 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 21:10:16.377941 13667 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 21:10:16.379187 13667 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39903
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790284076800039-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790284076800039-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46431,127.0.98.61:39919,127.0.98.60:44849
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790284076800039-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46431
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790284076800039-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13667
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790284076800039-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 19:49:06 UTC on ccycloud.quasar-vhqjoc.root.comops.site
build id 4670864
I20260924 21:10:16.379413 13667 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 21:10:16.379590 13667 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 21:10:16.383860 13675 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 21:10:16.383876 13672 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 21:10:16.383972 13667 server_base.cc:1034] running on GCE node
W20260924 21:10:16.383860 13673 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 21:10:16.384277 13667 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 21:10:16.384526 13667 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 21:10:16.385684 13667 hybrid_clock.cc:648] HybridClock initialized: now 1790284216385667 us; error 26 us; skew 500 ppm
Sep 24 21:10:16 dist-test-slave-c7tk krb5kdc[13654](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284216, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 21:10:16.395761 13667 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 21:10:16.396147 13667 webserver.cc:460] Webserver started at http://127.0.98.62:44083/ using document root <none> and password file <none>
I20260924 21:10:16.396306 13667 fs_manager.cc:359] Metadata directory not provided
I20260924 21:10:16.396355 13667 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 21:10:16.396437 13667 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 21:10:16.397238 13667 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790284076800039-392-0/minicluster-data/master-0/data/instance:
uuid: "aabb111c8a6b4665b10dd7adc0faa02e"
format_stamp: "Formatted at 2026-09-24 21:10:16 on dist-test-slave-c7tk"
server_key: "5b357fd205272c2db86c2f074acf189c"
server_key_iv: "1b3b1a4fbd515719faa03f3b98dddcf1"
server_key_version: "encryptionkey@0"
I20260924 21:10:16.397504 13667 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790284076800039-392-0/minicluster-data/master-0/wal/instance:
uuid: "aabb111c8a6b4665b10dd7adc0faa02e"
format_stamp: "Formatted at 2026-09-24 21:10:16 on dist-test-slave-c7tk"
server_key: "5b357fd205272c2db86c2f074acf189c"
server_key_iv: "1b3b1a4fbd515719faa03f3b98dddcf1"
server_key_version: "encryptionkey@0"
I20260924 21:10:16.398639 13667 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20260924 21:10:16.399288 13682 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 21:10:16.399473 13667 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 21:10:16.399559 13667 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790284076800039-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790284076800039-392-0/minicluster-data/master-0/wal
uuid: "aabb111c8a6b4665b10dd7adc0faa02e"
format_stamp: "Formatted at 2026-09-24 21:10:16 on dist-test-slave-c7tk"
server_key: "5b357fd205272c2db86c2f074acf189c"
server_key_iv: "1b3b1a4fbd515719faa03f3b98dddcf1"
server_key_version: "encryptionkey@0"
I20260924 21:10:16.399626 13667 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790284076800039-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790284076800039-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790284076800039-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 21:10:16.409683 13667 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 21:10:16.411304 13667 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 21:10:16.411444 13667 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 21:10:16.414690 13667 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:46431
I20260924 21:10:16.414731 13734 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:46431 every 8 connection(s)
I20260924 21:10:16.415011 13667 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790284076800039-392-0/minicluster-data/master-0/data/info.pb
I20260924 21:10:16.415702 13735 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 21:10:16.416960 13735 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46431 } has no permanent_uuid. Determining permanent_uuid...
I20260924 21:10:16.418737 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAzNNjS/build/release/bin/kudu as pid 13667
I20260924 21:10:16.418813 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790284076800039-392-0/minicluster-data/master-0/wal/instance
I20260924 21:10:16.419013 392 external_mini_cluster.cc:1428] Setting key 711f55f82f0d06079246052d60e532b6
Sep 24 21:10:16 dist-test-slave-c7tk krb5kdc[13654](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284216, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
I20260924 21:10:16.426124 13737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:10:16.417477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51960 (local address 127.0.98.62:46431)
0924 21:10:16.417635 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0924 21:10:16.417642 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 21:10:16.417782 (+ 140us) server_negotiation.cc:407] Connection header received
0924 21:10:16.418268 (+ 486us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:10:16.418270 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:10:16.418294 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:10:16.418393 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:10:16.418923 (+ 530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:10:16.419704 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:10:16.420980 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:10:16.421280 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:10:16.421353 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:10:16.423220 (+ 1867us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 21:10:16.423226 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 21:10:16.423226 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 21:10:16.423237 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 21:10:16.424557 (+ 1320us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:10:16.424983 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:10:16.424987 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:10:16.424988 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:10:16.425093 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:10:16.425387 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:10:16.425388 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:10:16.425388 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:10:16.425647 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 21:10:16.425723 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0924 21:10:16.425906 (+ 183us) server_negotiation.cc:299] Negotiation successful
0924 21:10:16.425982 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":71,"threads_started":1}
I20260924 21:10:16.426126 13738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:10:16.417460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46431 (local address 127.0.0.1:51960)
0924 21:10:16.417627 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0924 21:10:16.417662 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0924 21:10:16.418131 (+ 469us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 21:10:16.418399 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 21:10:16.418401 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 21:10:16.418651 (+ 250us) client_negotiation.cc:189] Negotiated authn=SASL
0924 21:10:16.418799 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:10:16.418805 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:10:16.419797 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:10:16.419798 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:10:16.420885 (+ 1087us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:10:16.420888 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:10:16.421363 (+ 475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:10:16.421364 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:10:16.421439 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:10:16.421820 (+ 381us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 21:10:16.421829 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 21:10:16.423125 (+ 1296us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 21:10:16.424643 (+ 1518us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 21:10:16.424648 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 21:10:16.424649 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 21:10:16.424872 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 21:10:16.425183 (+ 311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 21:10:16.425184 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 21:10:16.425184 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 21:10:16.425303 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 21:10:16.425706 (+ 403us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 21:10:16.425711 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 21:10:16.425759 (+ 48us) client_negotiation.cc:769] Sending connection context
0924 21:10:16.425872 (+ 113us) client_negotiation.cc:240] Negotiation successful
0924 21:10:16.426029 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"mutex_wait_us":50,"thread_start_us":81,"threads_started":1}
I20260924 21:10:16.426780 13735 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39919 } has no permanent_uuid. Determining permanent_uuid...
I20260924 21:10:16.427042 13738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:10:16.426957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:39919 (local address 127.0.0.1:33604)
0924 21:10:16.426994 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0924 21:10:16.427026 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:39919: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260924 21:10:16.427155 13685 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:39919: connect: Connection refused (error 111)
I20260924 21:10:16.427481 13738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:10:16.427409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:39919 (local address 127.0.0.1:33608)
0924 21:10:16.427447 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0924 21:10:16.427467 (+ 20us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:39919: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260924 21:10:16.427580 13735 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:39919: Network error: Client connection negotiation failed: client connection to 127.0.98.61:39919: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790284076800039-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790284076800039-392-0/krb5kdc/kudu_127.0.98.61.keytab.
I20260924 21:10:16.440363 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAzNNjS/build/release/bin/kudu
/tmp/dist-test-taskAzNNjS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790284076800039-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790284076800039-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790284076800039-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790284076800039-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:39919
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790284076800039-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39903
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46431,127.0.98.61:39919,127.0.98.60:44849
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790284076800039-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790284076800039-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790284076800039-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 21:10:16.476753 13735 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39919 } attempt: 1
I20260924 21:10:16.477079 13738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:10:16.476979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:39919 (local address 127.0.0.1:33624)
0924 21:10:16.477026 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0924 21:10:16.477059 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:39919: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260924 21:10:16.477334 13738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:10:16.477259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:39919 (local address 127.0.0.1:33628)
0924 21:10:16.477294 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0924 21:10:16.477313 (+ 19us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:39919: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
W20260924 21:10:16.477411 13735 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:39919: Network error: Client connection negotiation failed: client connection to 127.0.98.61:39919: connect: Connection refused (error 111)
W20260924 21:10:16.509342 13742 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 21:10:16.509498 13742 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 21:10:16.509516 13742 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 21:10:16.510643 13742 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 21:10:16.510682 13742 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 21:10:16.510696 13742 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 21:10:16.510710 13742 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 21:10:16.510720 13742 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 21:10:16.512023 13742 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39903
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790284076800039-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790284076800039-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46431,127.0.98.61:39919,127.0.98.60:44849
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790284076800039-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:39919
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790284076800039-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13742
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790284076800039-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 19:49:06 UTC on ccycloud.quasar-vhqjoc.root.comops.site
build id 4670864
I20260924 21:10:16.512231 13742 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 21:10:16.512415 13742 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 21:10:16.516659 13747 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 21:10:16.516665 13748 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 21:10:16.516665 13750 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 21:10:16.516804 13742 server_base.cc:1034] running on GCE node
I20260924 21:10:16.517043 13742 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 21:10:16.517275 13742 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 21:10:16.518427 13742 hybrid_clock.cc:648] HybridClock initialized: now 1790284216518418 us; error 32 us; skew 500 ppm
Sep 24 21:10:16 dist-test-slave-c7tk krb5kdc[13654](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284216, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 21:10:16.520504 13742 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260924 21:10:16.520951 13742 webserver.cc:460] Webserver started at http://127.0.98.61:42025/ using document root <none> and password file <none>
I20260924 21:10:16.521183 13742 fs_manager.cc:359] Metadata directory not provided
I20260924 21:10:16.521238 13742 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 21:10:16.521350 13742 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 21:10:16.522246 13742 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790284076800039-392-0/minicluster-data/master-1/data/instance:
uuid: "b79e00105bd14bee96da3db6960088c0"
format_stamp: "Formatted at 2026-09-24 21:10:16 on dist-test-slave-c7tk"
server_key: "4d2d0df89993299d97aa2a0a5bdb36fc"
server_key_iv: "bee9dd94eca19634910ee1e5f7c98697"
server_key_version: "encryptionkey@0"
I20260924 21:10:16.522541 13742 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790284076800039-392-0/minicluster-data/master-1/wal/instance:
uuid: "b79e00105bd14bee96da3db6960088c0"
format_stamp: "Formatted at 2026-09-24 21:10:16 on dist-test-slave-c7tk"
server_key: "4d2d0df89993299d97aa2a0a5bdb36fc"
server_key_iv: "bee9dd94eca19634910ee1e5f7c98697"
server_key_version: "encryptionkey@0"
I20260924 21:10:16.523725 13742 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 21:10:16.524510 13757 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 21:10:16.524667 13742 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 21:10:16.524755 13742 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790284076800039-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790284076800039-392-0/minicluster-data/master-1/wal
uuid: "b79e00105bd14bee96da3db6960088c0"
format_stamp: "Formatted at 2026-09-24 21:10:16 on dist-test-slave-c7tk"
server_key: "4d2d0df89993299d97aa2a0a5bdb36fc"
server_key_iv: "bee9dd94eca19634910ee1e5f7c98697"
server_key_version: "encryptionkey@0"
I20260924 21:10:16.524823 13742 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790284076800039-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790284076800039-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790284076800039-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 21:10:16.533792 13742 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 21:10:16.535211 13742 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 21:10:16.535363 13742 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 21:10:16.538756 13742 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:39919
I20260924 21:10:16.538784 13809 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:39919 every 8 connection(s)
I20260924 21:10:16.539052 13742 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790284076800039-392-0/minicluster-data/master-1/data/info.pb
I20260924 21:10:16.539700 13810 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 21:10:16.540990 13810 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46431 } has no permanent_uuid. Determining permanent_uuid...
I20260924 21:10:16.543844 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAzNNjS/build/release/bin/kudu as pid 13742
I20260924 21:10:16.543915 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790284076800039-392-0/minicluster-data/master-1/wal/instance
I20260924 21:10:16.544067 392 external_mini_cluster.cc:1428] Setting key 670727d2b3b903b7bd80002071f11cd6
I20260924 21:10:16.545534 13735 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39919 } attempt: 2
Sep 24 21:10:16 dist-test-slave-c7tk krb5kdc[13654](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284216, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Sep 24 21:10:16 dist-test-slave-c7tk krb5kdc[13654](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284216, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260924 21:10:16.550232 13737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:10:16.541590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51962 (local address 127.0.98.62:46431)
0924 21:10:16.541662 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0924 21:10:16.541664 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 21:10:16.541790 (+ 126us) server_negotiation.cc:407] Connection header received
0924 21:10:16.542274 (+ 484us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:10:16.542275 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:10:16.542283 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:10:16.542355 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:10:16.542836 (+ 481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:10:16.543298 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:10:16.544169 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:10:16.544363 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:10:16.544445 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:10:16.546441 (+ 1996us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 21:10:16.546442 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 21:10:16.546443 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 21:10:16.546449 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 21:10:16.548621 (+ 2172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:10:16.549093 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:10:16.549095 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:10:16.549095 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:10:16.549151 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:10:16.549426 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:10:16.549427 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:10:16.549427 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:10:16.549559 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 21:10:16.549623 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0924 21:10:16.549988 (+ 365us) server_negotiation.cc:299] Negotiation successful
0924 21:10:16.550131 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260924 21:10:16.550321 13812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:10:16.541559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46431 (local address 127.0.0.1:51962)
0924 21:10:16.541683 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0924 21:10:16.541716 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0924 21:10:16.542192 (+ 476us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 21:10:16.542399 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 21:10:16.542403 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 21:10:16.542606 (+ 203us) client_negotiation.cc:189] Negotiated authn=SASL
0924 21:10:16.542772 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:10:16.542775 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:10:16.543363 (+ 588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:10:16.543364 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:10:16.544105 (+ 741us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:10:16.544107 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:10:16.544453 (+ 346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:10:16.544454 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:10:16.544563 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:10:16.544958 (+ 395us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 21:10:16.544964 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0924 21:10:16.546366 (+ 1402us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 21:10:16.548723 (+ 2357us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 21:10:16.548730 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 21:10:16.548732 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 21:10:16.549015 (+ 283us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 21:10:16.549227 (+ 212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 21:10:16.549228 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 21:10:16.549229 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 21:10:16.549352 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 21:10:16.549630 (+ 278us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 21:10:16.549636 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 21:10:16.549774 (+ 138us) client_negotiation.cc:769] Sending connection context
0924 21:10:16.549956 (+ 182us) client_negotiation.cc:240] Negotiation successful
0924 21:10:16.550101 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":91,"thread_start_us":63,"threads_started":1}
I20260924 21:10:16.550987 13810 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39919 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
I20260924 21:10:16.553128 13738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:10:16.545745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:39919 (local address 127.0.0.1:33644)
0924 21:10:16.545799 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0924 21:10:16.545834 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0924 21:10:16.545870 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 21:10:16.546065 (+ 195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 21:10:16.546066 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 21:10:16.546191 (+ 125us) client_negotiation.cc:189] Negotiated authn=SASL
0924 21:10:16.546229 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:10:16.546230 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:10:16.547299 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:10:16.547300 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:10:16.547826 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:10:16.547828 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:10:16.548252 (+ 424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:10:16.548253 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:10:16.548328 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:10:16.548713 (+ 385us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 21:10:16.548718 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 21:10:16.550137 (+ 1419us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 21:10:16.552201 (+ 2064us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 21:10:16.552201 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 21:10:16.552202 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 21:10:16.552348 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 21:10:16.552549 (+ 201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 21:10:16.552549 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 21:10:16.552550 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 21:10:16.552598 (+ 48us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 21:10:16.552847 (+ 249us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 21:10:16.552848 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 21:10:16.552883 (+ 35us) client_negotiation.cc:769] Sending connection context
0924 21:10:16.552944 (+ 61us) client_negotiation.cc:240] Negotiation successful
0924 21:10:16.553021 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40}
I20260924 21:10:16.553133 13814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:10:16.545776 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33644 (local address 127.0.98.61:39919)
0924 21:10:16.545948 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0924 21:10:16.545952 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 21:10:16.545967 (+ 15us) server_negotiation.cc:407] Connection header received
0924 21:10:16.545985 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:10:16.545987 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:10:16.546003 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:10:16.546062 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:10:16.546306 (+ 244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:10:16.547193 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:10:16.547910 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:10:16.548154 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:10:16.548252 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:10:16.550221 (+ 1969us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 21:10:16.550226 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 21:10:16.550226 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 21:10:16.550232 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 21:10:16.552098 (+ 1866us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:10:16.552434 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:10:16.552437 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:10:16.552437 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:10:16.552489 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:10:16.552662 (+ 173us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:10:16.552663 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:10:16.552663 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:10:16.552798 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 21:10:16.552862 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0924 21:10:16.552969 (+ 107us) server_negotiation.cc:299] Negotiation successful
0924 21:10:16.553026 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":152,"thread_start_us":82,"threads_started":1}
I20260924 21:10:16.553694 13735 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44849 } has no permanent_uuid. Determining permanent_uuid...
I20260924 21:10:16.554070 13738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:10:16.553937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44849 (local address 127.0.0.1:55230)
0924 21:10:16.554021 (+ 84us) negotiation.cc:107] Waiting for socket to connect
0924 21:10:16.554047 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44849: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
I20260924 21:10:16.554375 13738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:10:16.554299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44849 (local address 127.0.0.1:55240)
0924 21:10:16.554332 (+ 33us) negotiation.cc:107] Waiting for socket to connect
0924 21:10:16.554352 (+ 20us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44849: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
W20260924 21:10:16.554450 13735 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:44849: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44849: connect: Connection refused (error 111)
Sep 24 21:10:16 dist-test-slave-c7tk krb5kdc[13654](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284216, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260924 21:10:16.557633 13812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:10:16.551264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:39919 (local address 127.0.0.1:33650)
0924 21:10:16.551293 (+ 29us) negotiation.cc:107] Waiting for socket to connect
0924 21:10:16.551301 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0924 21:10:16.551340 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 21:10:16.551540 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 21:10:16.551540 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 21:10:16.551652 (+ 112us) client_negotiation.cc:189] Negotiated authn=SASL
0924 21:10:16.551671 (+ 19us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:10:16.551672 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:10:16.552269 (+ 597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:10:16.552269 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:10:16.552657 (+ 388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:10:16.552658 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:10:16.553038 (+ 380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:10:16.553039 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:10:16.553095 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:10:16.553630 (+ 535us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 21:10:16.553635 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 21:10:16.555016 (+ 1381us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 21:10:16.556835 (+ 1819us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 21:10:16.556836 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 21:10:16.556836 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 21:10:16.556981 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 21:10:16.557147 (+ 166us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 21:10:16.557148 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 21:10:16.557148 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 21:10:16.557193 (+ 45us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 21:10:16.557387 (+ 194us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 21:10:16.557387 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 21:10:16.557418 (+ 31us) client_negotiation.cc:769] Sending connection context
0924 21:10:16.557469 (+ 51us) client_negotiation.cc:240] Negotiation successful
0924 21:10:16.557545 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16}
I20260924 21:10:16.557714 13815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:10:16.551325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33650 (local address 127.0.98.61:39919)
0924 21:10:16.551448 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0924 21:10:16.551450 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 21:10:16.551458 (+ 8us) server_negotiation.cc:407] Connection header received
0924 21:10:16.551474 (+ 16us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:10:16.551475 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:10:16.551482 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:10:16.551539 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:10:16.551733 (+ 194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:10:16.552193 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:10:16.552733 (+ 540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:10:16.552964 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:10:16.553037 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:10:16.555098 (+ 2061us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 21:10:16.555099 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 21:10:16.555100 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 21:10:16.555105 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 21:10:16.556741 (+ 1636us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:10:16.557041 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:10:16.557042 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:10:16.557043 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:10:16.557091 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:10:16.557242 (+ 151us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:10:16.557242 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:10:16.557242 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:10:16.557325 (+ 83us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 21:10:16.557385 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0924 21:10:16.557485 (+ 100us) server_negotiation.cc:299] Negotiation successful
0924 21:10:16.557589 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":51,"server-negotiator.queue_time_us":106,"thread_start_us":88,"threads_started":1}
I20260924 21:10:16.558148 13810 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44849 } has no permanent_uuid. Determining permanent_uuid...
I20260924 21:10:16.558409 13812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:10:16.558332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44849 (local address 127.0.0.1:55246)
0924 21:10:16.558365 (+ 33us) negotiation.cc:107] Waiting for socket to connect
0924 21:10:16.558395 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44849: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":16}
W20260924 21:10:16.558542 13761 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44849: connect: Connection refused (error 111)
I20260924 21:10:16.558880 13812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:10:16.558805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44849 (local address 127.0.0.1:55260)
0924 21:10:16.558847 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0924 21:10:16.558867 (+ 20us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44849: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260924 21:10:16.558995 13810 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:44849: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44849: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790284076800039-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790284076800039-392-0/krb5kdc/kudu_127.0.98.60.keytab.
I20260924 21:10:16.566910 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAzNNjS/build/release/bin/kudu
/tmp/dist-test-taskAzNNjS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790284076800039-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790284076800039-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790284076800039-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790284076800039-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:44849
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790284076800039-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39903
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46431,127.0.98.61:39919,127.0.98.60:44849
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790284076800039-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790284076800039-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790284076800039-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 21:10:16.597586 13735 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44849 } attempt: 1
I20260924 21:10:16.597931 13738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:10:16.597821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44849 (local address 127.0.0.1:55262)
0924 21:10:16.597869 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0924 21:10:16.597903 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44849: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
I20260924 21:10:16.598241 13738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:10:16.598164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44849 (local address 127.0.0.1:55276)
0924 21:10:16.598202 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0924 21:10:16.598225 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44849: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260924 21:10:16.598330 13735 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:44849: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44849: connect: Connection refused (error 111)
I20260924 21:10:16.608248 13810 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44849 } attempt: 1
I20260924 21:10:16.608582 13812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:10:16.608487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44849 (local address 127.0.0.1:55292)
0924 21:10:16.608537 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0924 21:10:16.608564 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44849: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260924 21:10:16.608875 13812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:10:16.608811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44849 (local address 127.0.0.1:55302)
0924 21:10:16.608843 (+ 32us) negotiation.cc:107] Waiting for socket to connect
0924 21:10:16.608863 (+ 20us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44849: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
W20260924 21:10:16.608970 13810 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:44849: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44849: connect: Connection refused (error 111)
W20260924 21:10:16.641271 13818 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 21:10:16.641424 13818 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 21:10:16.641443 13818 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 21:10:16.642417 13818 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 21:10:16.642448 13818 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 21:10:16.642460 13818 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 21:10:16.642470 13818 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 21:10:16.642488 13818 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 21:10:16.643764 13818 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39903
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790284076800039-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790284076800039-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46431,127.0.98.61:39919,127.0.98.60:44849
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790284076800039-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:44849
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790284076800039-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13818
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790284076800039-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 19:49:06 UTC on ccycloud.quasar-vhqjoc.root.comops.site
build id 4670864
I20260924 21:10:16.643934 13818 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 21:10:16.644059 13818 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 21:10:16.645453 13735 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44849 } attempt: 2
I20260924 21:10:16.645857 13738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:10:16.645699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44849 (local address 127.0.0.1:55310)
0924 21:10:16.645756 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0924 21:10:16.645790 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44849: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260924 21:10:16.646160 13738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:10:16.646064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44849 (local address 127.0.0.1:55322)
0924 21:10:16.646105 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0924 21:10:16.646131 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44849: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260924 21:10:16.646246 13735 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:44849: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44849: connect: Connection refused (error 111)
W20260924 21:10:16.648936 13823 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 21:10:16.648898 13824 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 21:10:16.648903 13826 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 21:10:16.649003 13818 server_base.cc:1034] running on GCE node
I20260924 21:10:16.649319 13818 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 21:10:16.649519 13818 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 21:10:16.650697 13818 hybrid_clock.cc:648] HybridClock initialized: now 1790284216650687 us; error 22 us; skew 500 ppm
Sep 24 21:10:16 dist-test-slave-c7tk krb5kdc[13654](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284216, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 21:10:16.652722 13818 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260924 21:10:16.653149 13818 webserver.cc:460] Webserver started at http://127.0.98.60:40575/ using document root <none> and password file <none>
I20260924 21:10:16.653359 13818 fs_manager.cc:359] Metadata directory not provided
I20260924 21:10:16.653424 13818 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 21:10:16.653563 13818 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 21:10:16.654554 13818 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790284076800039-392-0/minicluster-data/master-2/data/instance:
uuid: "8ccf50cb37604a33a3099ecee0355f0d"
format_stamp: "Formatted at 2026-09-24 21:10:16 on dist-test-slave-c7tk"
server_key: "d414e7caa1246867226302c938a1457b"
server_key_iv: "17c7ece64376ed7ddd52cb546407022f"
server_key_version: "encryptionkey@0"
I20260924 21:10:16.654930 13818 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790284076800039-392-0/minicluster-data/master-2/wal/instance:
uuid: "8ccf50cb37604a33a3099ecee0355f0d"
format_stamp: "Formatted at 2026-09-24 21:10:16 on dist-test-slave-c7tk"
server_key: "d414e7caa1246867226302c938a1457b"
server_key_iv: "17c7ece64376ed7ddd52cb546407022f"
server_key_version: "encryptionkey@0"
I20260924 21:10:16.656064 13818 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260924 21:10:16.656704 13833 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 21:10:16.656844 13818 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 21:10:16.656926 13818 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790284076800039-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790284076800039-392-0/minicluster-data/master-2/wal
uuid: "8ccf50cb37604a33a3099ecee0355f0d"
format_stamp: "Formatted at 2026-09-24 21:10:16 on dist-test-slave-c7tk"
server_key: "d414e7caa1246867226302c938a1457b"
server_key_iv: "17c7ece64376ed7ddd52cb546407022f"
server_key_version: "encryptionkey@0"
I20260924 21:10:16.656997 13818 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790284076800039-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790284076800039-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790284076800039-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 21:10:16.677188 13810 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44849 } attempt: 2
I20260924 21:10:16.677561 13812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:10:16.677458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44849 (local address 127.0.0.1:55330)
0924 21:10:16.677507 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0924 21:10:16.677536 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44849: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260924 21:10:16.677910 13812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:10:16.677825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44849 (local address 127.0.0.1:55344)
0924 21:10:16.677868 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0924 21:10:16.677889 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44849: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":19}
W20260924 21:10:16.677997 13810 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:44849: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44849: connect: Connection refused (error 111)
I20260924 21:10:16.678711 13818 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 21:10:16.680723 13818 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 21:10:16.680878 13818 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 21:10:16.684350 13818 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:44849
I20260924 21:10:16.684401 13885 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:44849 every 8 connection(s)
I20260924 21:10:16.684689 13818 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790284076800039-392-0/minicluster-data/master-2/data/info.pb
I20260924 21:10:16.685441 13886 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 21:10:16.686692 13886 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46431 } has no permanent_uuid. Determining permanent_uuid...
I20260924 21:10:16.690587 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAzNNjS/build/release/bin/kudu as pid 13818
I20260924 21:10:16.690663 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790284076800039-392-0/minicluster-data/master-2/wal/instance
I20260924 21:10:16.690817 392 external_mini_cluster.cc:1428] Setting key fe3ecde08b0e424d084928e3128b6f51
Sep 24 21:10:16 dist-test-slave-c7tk krb5kdc[13654](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284216, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 21:10:16.694742 13737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:10:16.687250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51974 (local address 127.0.98.62:46431)
0924 21:10:16.687301 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0924 21:10:16.687303 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 21:10:16.687463 (+ 160us) server_negotiation.cc:407] Connection header received
0924 21:10:16.687869 (+ 406us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:10:16.687870 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:10:16.687880 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:10:16.687944 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:10:16.688430 (+ 486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:10:16.688834 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:10:16.689652 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:10:16.689872 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:10:16.689976 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:10:16.692199 (+ 2223us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 21:10:16.692200 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 21:10:16.692200 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 21:10:16.692208 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 21:10:16.693431 (+ 1223us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:10:16.693828 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:10:16.693829 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:10:16.693829 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:10:16.693893 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:10:16.694143 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:10:16.694144 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:10:16.694144 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:10:16.694266 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 21:10:16.694356 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0924 21:10:16.694590 (+ 234us) server_negotiation.cc:299] Negotiation successful
0924 21:10:16.694670 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260924 21:10:16.694875 13888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:10:16.687226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46431 (local address 127.0.0.1:51974)
0924 21:10:16.687357 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0924 21:10:16.687384 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0924 21:10:16.687779 (+ 395us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 21:10:16.688003 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 21:10:16.688006 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 21:10:16.688193 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0924 21:10:16.688343 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:10:16.688346 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:10:16.688919 (+ 573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:10:16.688920 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:10:16.689570 (+ 650us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:10:16.689572 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:10:16.689982 (+ 410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:10:16.689983 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:10:16.690107 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:10:16.690549 (+ 442us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 21:10:16.690556 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0924 21:10:16.692120 (+ 1564us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 21:10:16.693513 (+ 1393us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 21:10:16.693519 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 21:10:16.693521 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 21:10:16.693732 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 21:10:16.693961 (+ 229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 21:10:16.693961 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 21:10:16.693962 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 21:10:16.694074 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 21:10:16.694341 (+ 267us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 21:10:16.694346 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 21:10:16.694469 (+ 123us) client_negotiation.cc:769] Sending connection context
0924 21:10:16.694558 (+ 89us) client_negotiation.cc:240] Negotiation successful
0924 21:10:16.694688 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":66,"threads_started":1}
I20260924 21:10:16.695289 13886 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39919 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 21:10:16 dist-test-slave-c7tk krb5kdc[13654](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284216, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Sep 24 21:10:16 dist-test-slave-c7tk krb5kdc[13654](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284216, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260924 21:10:16.698710 13890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:10:16.691253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55348 (local address 127.0.98.60:44849)
0924 21:10:16.691399 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0924 21:10:16.691403 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 21:10:16.691420 (+ 17us) server_negotiation.cc:407] Connection header received
0924 21:10:16.691465 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:10:16.691467 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:10:16.691485 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:10:16.691582 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:10:16.692128 (+ 546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:10:16.692970 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:10:16.693715 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:10:16.694097 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:10:16.694185 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:10:16.696202 (+ 2017us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 21:10:16.696209 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 21:10:16.696209 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 21:10:16.696215 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 21:10:16.697563 (+ 1348us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:10:16.697898 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:10:16.697900 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:10:16.697901 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:10:16.697958 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:10:16.698145 (+ 187us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:10:16.698146 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:10:16.698146 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:10:16.698264 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 21:10:16.698432 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0924 21:10:16.698471 (+ 39us) server_negotiation.cc:299] Negotiation successful
0924 21:10:16.698537 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121,"thread_start_us":61,"threads_started":1}
I20260924 21:10:16.701346 13888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:10:16.695485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:39919 (local address 127.0.0.1:33664)
0924 21:10:16.695528 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0924 21:10:16.695538 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 21:10:16.695573 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 21:10:16.695675 (+ 102us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 21:10:16.695676 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 21:10:16.695789 (+ 113us) client_negotiation.cc:189] Negotiated authn=SASL
0924 21:10:16.695824 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:10:16.695825 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:10:16.696400 (+ 575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:10:16.696401 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:10:16.696817 (+ 416us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:10:16.696818 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:10:16.697167 (+ 349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:10:16.697168 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:10:16.697225 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:10:16.697602 (+ 377us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 21:10:16.697606 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 21:10:16.698712 (+ 1106us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 21:10:16.700412 (+ 1700us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 21:10:16.700412 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 21:10:16.700412 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 21:10:16.700558 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 21:10:16.700760 (+ 202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 21:10:16.700760 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 21:10:16.700760 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 21:10:16.700805 (+ 45us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 21:10:16.701094 (+ 289us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 21:10:16.701095 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 21:10:16.701131 (+ 36us) client_negotiation.cc:769] Sending connection context
0924 21:10:16.701201 (+ 70us) client_negotiation.cc:240] Negotiation successful
0924 21:10:16.701272 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29}
I20260924 21:10:16.701467 13815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:10:16.695545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33664 (local address 127.0.98.61:39919)
0924 21:10:16.695578 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0924 21:10:16.695580 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 21:10:16.695589 (+ 9us) server_negotiation.cc:407] Connection header received
0924 21:10:16.695622 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:10:16.695623 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:10:16.695630 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:10:16.695665 (+ 35us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:10:16.695900 (+ 235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:10:16.696307 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:10:16.696908 (+ 601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:10:16.697095 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:10:16.697167 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:10:16.698813 (+ 1646us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 21:10:16.698814 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 21:10:16.698815 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 21:10:16.698828 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0924 21:10:16.700313 (+ 1485us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:10:16.700625 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:10:16.700626 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:10:16.700627 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:10:16.700693 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:10:16.700897 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:10:16.700898 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:10:16.700898 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:10:16.701015 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 21:10:16.701083 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0924 21:10:16.701217 (+ 134us) server_negotiation.cc:299] Negotiation successful
0924 21:10:16.701324 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260924 21:10:16.701802 13886 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44849 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 21:10:16 dist-test-slave-c7tk krb5kdc[13654](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284216, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Sep 24 21:10:16 dist-test-slave-c7tk krb5kdc[13654](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284216, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260924 21:10:16.706943 13737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:10:16.699367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51986 (local address 127.0.98.62:46431)
0924 21:10:16.699399 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0924 21:10:16.699401 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 21:10:16.699587 (+ 186us) server_negotiation.cc:407] Connection header received
0924 21:10:16.699639 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:10:16.699640 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:10:16.699649 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:10:16.699721 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:10:16.700155 (+ 434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:10:16.700775 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:10:16.701750 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:10:16.702032 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:10:16.702091 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:10:16.704425 (+ 2334us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 21:10:16.704426 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 21:10:16.704426 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 21:10:16.704433 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 21:10:16.705842 (+ 1409us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:10:16.706212 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:10:16.706213 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:10:16.706213 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:10:16.706262 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:10:16.706472 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:10:16.706473 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:10:16.706473 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:10:16.706574 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 21:10:16.706763 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0924 21:10:16.706800 (+ 37us) server_negotiation.cc:299] Negotiation successful
0924 21:10:16.706863 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260924 21:10:16.708499 13888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:10:16.702024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44849 (local address 127.0.0.1:55350)
0924 21:10:16.702057 (+ 33us) negotiation.cc:107] Waiting for socket to connect
0924 21:10:16.702066 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 21:10:16.702126 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 21:10:16.703178 (+ 1052us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 21:10:16.703180 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 21:10:16.703318 (+ 138us) client_negotiation.cc:189] Negotiated authn=SASL
0924 21:10:16.703349 (+ 31us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:10:16.703351 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:10:16.703931 (+ 580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:10:16.703932 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:10:16.704387 (+ 455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:10:16.704388 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:10:16.704819 (+ 431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:10:16.704826 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:10:16.704880 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:10:16.705152 (+ 272us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 21:10:16.705157 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 21:10:16.706287 (+ 1130us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 21:10:16.707635 (+ 1348us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 21:10:16.707636 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 21:10:16.707636 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 21:10:16.707799 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 21:10:16.708003 (+ 204us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 21:10:16.708004 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 21:10:16.708004 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 21:10:16.708081 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 21:10:16.708329 (+ 248us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 21:10:16.708330 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 21:10:16.708357 (+ 27us) client_negotiation.cc:769] Sending connection context
0924 21:10:16.708403 (+ 46us) client_negotiation.cc:240] Negotiation successful
0924 21:10:16.708455 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19}
I20260924 21:10:16.708599 13890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:10:16.703012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55350 (local address 127.0.98.60:44849)
0924 21:10:16.703072 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0924 21:10:16.703074 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 21:10:16.703081 (+ 7us) server_negotiation.cc:407] Connection header received
0924 21:10:16.703096 (+ 15us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:10:16.703097 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:10:16.703104 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:10:16.703155 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:10:16.703419 (+ 264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:10:16.703843 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:10:16.704473 (+ 630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:10:16.704747 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:10:16.704803 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:10:16.706370 (+ 1567us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 21:10:16.706371 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 21:10:16.706372 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 21:10:16.706377 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 21:10:16.707565 (+ 1188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:10:16.707881 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:10:16.707882 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:10:16.707882 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:10:16.707939 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:10:16.708144 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:10:16.708145 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:10:16.708145 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:10:16.708263 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 21:10:16.708333 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0924 21:10:16.708430 (+ 97us) server_negotiation.cc:299] Negotiation successful
0924 21:10:16.708517 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":10,"server-negotiator.queue_time_us":46}
I20260924 21:10:16.709708 13886 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8ccf50cb37604a33a3099ecee0355f0d: Bootstrap starting.
I20260924 21:10:16.710721 13886 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8ccf50cb37604a33a3099ecee0355f0d: Neither blocks nor log segments found. Creating new log.
I20260924 21:10:16.711069 13886 log.cc:824] T 00000000000000000000000000000000 P 8ccf50cb37604a33a3099ecee0355f0d: Log is configured to *not* fsync() on all Append() calls
I20260924 21:10:16.711781 13886 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8ccf50cb37604a33a3099ecee0355f0d: No bootstrap required, opened a new log
Sep 24 21:10:16 dist-test-slave-c7tk krb5kdc[13654](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284216, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260924 21:10:16.713581 13886 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8ccf50cb37604a33a3099ecee0355f0d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aabb111c8a6b4665b10dd7adc0faa02e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46431 } } peers { permanent_uuid: "b79e00105bd14bee96da3db6960088c0" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39919 } } peers { permanent_uuid: "8ccf50cb37604a33a3099ecee0355f0d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44849 } }
I20260924 21:10:16.713721 13886 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8ccf50cb37604a33a3099ecee0355f0d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 21:10:16.713770 13886 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8ccf50cb37604a33a3099ecee0355f0d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ccf50cb37604a33a3099ecee0355f0d, State: Initialized, Role: FOLLOWER
I20260924 21:10:16.713949 13886 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8ccf50cb37604a33a3099ecee0355f0d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aabb111c8a6b4665b10dd7adc0faa02e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46431 } } peers { permanent_uuid: "b79e00105bd14bee96da3db6960088c0" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39919 } } peers { permanent_uuid: "8ccf50cb37604a33a3099ecee0355f0d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44849 } }
I20260924 21:10:16.714258 13886 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8ccf50cb37604a33a3099ecee0355f0d [sys.catalog]: configured and running, proceeding with master startup.
I20260924 21:10:16.714234 13892 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8ccf50cb37604a33a3099ecee0355f0d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aabb111c8a6b4665b10dd7adc0faa02e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46431 } } peers { permanent_uuid: "b79e00105bd14bee96da3db6960088c0" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39919 } } peers { permanent_uuid: "8ccf50cb37604a33a3099ecee0355f0d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44849 } } }
I20260924 21:10:16.714324 13892 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8ccf50cb37604a33a3099ecee0355f0d [sys.catalog]: This master's current role is: FOLLOWER
I20260924 21:10:16.714754 13815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:10:16.707923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33668 (local address 127.0.98.61:39919)
0924 21:10:16.707951 (+ 28us) server_negotiation.cc:206] Beginning negotiation
0924 21:10:16.707953 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 21:10:16.708074 (+ 121us) server_negotiation.cc:407] Connection header received
0924 21:10:16.708095 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:10:16.708095 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:10:16.708103 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:10:16.708151 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:10:16.708678 (+ 527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:10:16.709146 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:10:16.709962 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:10:16.710126 (+ 164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:10:16.710177 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:10:16.712402 (+ 2225us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 21:10:16.712403 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 21:10:16.712404 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 21:10:16.712409 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 21:10:16.713638 (+ 1229us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:10:16.714027 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:10:16.714028 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:10:16.714028 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:10:16.714079 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:10:16.714286 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:10:16.714287 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:10:16.714287 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:10:16.714408 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 21:10:16.714513 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0924 21:10:16.714602 (+ 89us) server_negotiation.cc:299] Negotiation successful
0924 21:10:16.714647 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14}
W20260924 21:10:16.715981 13903 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 8ccf50cb37604a33a3099ecee0355f0d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 21:10:16.716063 13903 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 21:10:16.716216 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 21:10:16.716131 13818 master_runner.cc:186] Error getting master registration for 127.0.98.62:46431: OK, instance_id { permanent_uuid: "aabb111c8a6b4665b10dd7adc0faa02e" instance_seqno: 1790284216413725 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260924 21:10:16.718350 13818 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260924 21:10:16.726012 13737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:10:16.717628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51990 (local address 127.0.98.62:46431)
0924 21:10:16.717657 (+ 29us) server_negotiation.cc:206] Beginning negotiation
0924 21:10:16.717658 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 21:10:16.717749 (+ 91us) server_negotiation.cc:407] Connection header received
0924 21:10:16.717856 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:10:16.717857 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:10:16.717864 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:10:16.717889 (+ 25us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:10:16.718612 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:10:16.719205 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:10:16.720347 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:10:16.720623 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:10:16.720697 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:10:16.722228 (+ 1531us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 21:10:16.722229 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 21:10:16.722229 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 21:10:16.722236 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 21:10:16.723968 (+ 1732us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:10:16.724601 (+ 633us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:10:16.724603 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:10:16.724603 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:10:16.724660 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:10:16.725319 (+ 659us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:10:16.725333 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:10:16.725333 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:10:16.725468 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 21:10:16.725546 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0924 21:10:16.725793 (+ 247us) server_negotiation.cc:299] Negotiation successful
0924 21:10:16.725896 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14}
I20260924 21:10:16.726747 13815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:10:16.717668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33682 (local address 127.0.98.61:39919)
0924 21:10:16.717698 (+ 30us) server_negotiation.cc:206] Beginning negotiation
0924 21:10:16.717700 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 21:10:16.720204 (+ 2504us) server_negotiation.cc:407] Connection header received
0924 21:10:16.720295 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:10:16.720296 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:10:16.720305 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:10:16.720332 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:10:16.721024 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:10:16.721613 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:10:16.722569 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:10:16.722818 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:10:16.722854 (+ 36us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:10:16.723995 (+ 1141us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 21:10:16.723996 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 21:10:16.723996 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 21:10:16.724002 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 21:10:16.725522 (+ 1520us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:10:16.725952 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:10:16.725953 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:10:16.725953 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:10:16.726017 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:10:16.726278 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:10:16.726278 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:10:16.726279 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:10:16.726375 (+ 96us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 21:10:16.726452 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0924 21:10:16.726616 (+ 164us) server_negotiation.cc:299] Negotiation successful
0924 21:10:16.726678 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260924 21:10:16.728794 13890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:10:16.717860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55356 (local address 127.0.98.60:44849)
0924 21:10:16.718034 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0924 21:10:16.718036 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 21:10:16.722017 (+ 3981us) server_negotiation.cc:407] Connection header received
0924 21:10:16.722037 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:10:16.722038 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:10:16.722046 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:10:16.722090 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:10:16.722739 (+ 649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:10:16.723376 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:10:16.724240 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:10:16.724509 (+ 269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:10:16.724574 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:10:16.726072 (+ 1498us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 21:10:16.726073 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 21:10:16.726074 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 21:10:16.726080 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 21:10:16.727708 (+ 1628us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:10:16.728081 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:10:16.728082 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:10:16.728083 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:10:16.728132 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:10:16.728323 (+ 191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:10:16.728324 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:10:16.728324 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:10:16.728443 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 21:10:16.728550 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0924 21:10:16.728646 (+ 96us) server_negotiation.cc:299] Negotiation successful
0924 21:10:16.728708 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141}
I20260924 21:10:16.753388 13735 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44849 } attempt: 3
Sep 24 21:10:16 dist-test-slave-c7tk krb5kdc[13654](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284216, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260924 21:10:16.759734 13890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:10:16.753734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55366 (local address 127.0.98.60:44849)
0924 21:10:16.753814 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0924 21:10:16.753815 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 21:10:16.753821 (+ 6us) server_negotiation.cc:407] Connection header received
0924 21:10:16.753854 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:10:16.753855 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:10:16.753863 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:10:16.753916 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:10:16.754202 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:10:16.754604 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:10:16.755481 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:10:16.755628 (+ 147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:10:16.755681 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:10:16.757299 (+ 1618us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 21:10:16.757300 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 21:10:16.757300 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 21:10:16.757306 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 21:10:16.758685 (+ 1379us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:10:16.759051 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:10:16.759052 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:10:16.759052 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:10:16.759101 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:10:16.759278 (+ 177us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:10:16.759279 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:10:16.759279 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:10:16.759391 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 21:10:16.759465 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0924 21:10:16.759597 (+ 132us) server_negotiation.cc:299] Negotiation successful
0924 21:10:16.759667 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260924 21:10:16.759763 13738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:10:16.753658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44849 (local address 127.0.0.1:55366)
0924 21:10:16.753734 (+ 76us) negotiation.cc:107] Waiting for socket to connect
0924 21:10:16.753745 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 21:10:16.753787 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 21:10:16.753943 (+ 156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 21:10:16.753944 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 21:10:16.754098 (+ 154us) client_negotiation.cc:189] Negotiated authn=SASL
0924 21:10:16.754135 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:10:16.754136 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:10:16.754682 (+ 546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:10:16.754683 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:10:16.755381 (+ 698us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:10:16.755383 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:10:16.755699 (+ 316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:10:16.755700 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:10:16.755749 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:10:16.756060 (+ 311us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 21:10:16.756064 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 21:10:16.757213 (+ 1149us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 21:10:16.758769 (+ 1556us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 21:10:16.758770 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 21:10:16.758771 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 21:10:16.758969 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 21:10:16.759159 (+ 190us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 21:10:16.759159 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 21:10:16.759160 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 21:10:16.759207 (+ 47us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 21:10:16.759460 (+ 253us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 21:10:16.759461 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 21:10:16.759497 (+ 36us) client_negotiation.cc:769] Sending connection context
0924 21:10:16.759571 (+ 74us) client_negotiation.cc:240] Negotiation successful
0924 21:10:16.759663 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":57}
I20260924 21:10:16.761356 13735 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aabb111c8a6b4665b10dd7adc0faa02e: Bootstrap starting.
I20260924 21:10:16.761960 13735 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P aabb111c8a6b4665b10dd7adc0faa02e: Neither blocks nor log segments found. Creating new log.
I20260924 21:10:16.762204 13735 log.cc:824] T 00000000000000000000000000000000 P aabb111c8a6b4665b10dd7adc0faa02e: Log is configured to *not* fsync() on all Append() calls
I20260924 21:10:16.762825 13735 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aabb111c8a6b4665b10dd7adc0faa02e: No bootstrap required, opened a new log
I20260924 21:10:16.764154 13735 raft_consensus.cc:348] T 00000000000000000000000000000000 P aabb111c8a6b4665b10dd7adc0faa02e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aabb111c8a6b4665b10dd7adc0faa02e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46431 } } peers { permanent_uuid: "b79e00105bd14bee96da3db6960088c0" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39919 } } peers { permanent_uuid: "8ccf50cb37604a33a3099ecee0355f0d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44849 } }
I20260924 21:10:16.764266 13735 raft_consensus.cc:374] T 00000000000000000000000000000000 P aabb111c8a6b4665b10dd7adc0faa02e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 21:10:16.764307 13735 raft_consensus.cc:729] T 00000000000000000000000000000000 P aabb111c8a6b4665b10dd7adc0faa02e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aabb111c8a6b4665b10dd7adc0faa02e, State: Initialized, Role: FOLLOWER
I20260924 21:10:16.764415 13735 consensus_queue.cc:260] T 00000000000000000000000000000000 P aabb111c8a6b4665b10dd7adc0faa02e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aabb111c8a6b4665b10dd7adc0faa02e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46431 } } peers { permanent_uuid: "b79e00105bd14bee96da3db6960088c0" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39919 } } peers { permanent_uuid: "8ccf50cb37604a33a3099ecee0355f0d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44849 } }
I20260924 21:10:16.764672 13913 sys_catalog.cc:455] T 00000000000000000000000000000000 P aabb111c8a6b4665b10dd7adc0faa02e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aabb111c8a6b4665b10dd7adc0faa02e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46431 } } peers { permanent_uuid: "b79e00105bd14bee96da3db6960088c0" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39919 } } peers { permanent_uuid: "8ccf50cb37604a33a3099ecee0355f0d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44849 } } }
I20260924 21:10:16.764727 13735 sys_catalog.cc:564] T 00000000000000000000000000000000 P aabb111c8a6b4665b10dd7adc0faa02e [sys.catalog]: configured and running, proceeding with master startup.
I20260924 21:10:16.764741 13913 sys_catalog.cc:458] T 00000000000000000000000000000000 P aabb111c8a6b4665b10dd7adc0faa02e [sys.catalog]: This master's current role is: FOLLOWER
W20260924 21:10:16.766232 13924 catalog_manager.cc:1558] T 00000000000000000000000000000000 P aabb111c8a6b4665b10dd7adc0faa02e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 21:10:16.766278 13924 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 21:10:16.766819 13667 master_runner.cc:186] Error getting master registration for 127.0.98.61:39919: OK, instance_id { permanent_uuid: "b79e00105bd14bee96da3db6960088c0" instance_seqno: 1790284216537763 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260924 21:10:16.768712 13667 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260924 21:10:16.769157 13810 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44849 } attempt: 3
Sep 24 21:10:16 dist-test-slave-c7tk krb5kdc[13654](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284216, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260924 21:10:16.776234 13812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:10:16.769353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44849 (local address 127.0.0.1:55368)
0924 21:10:16.769428 (+ 75us) negotiation.cc:107] Waiting for socket to connect
0924 21:10:16.769437 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 21:10:16.769467 (+ 30us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 21:10:16.769686 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 21:10:16.769687 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 21:10:16.769868 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0924 21:10:16.769935 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:10:16.769937 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:10:16.770800 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:10:16.770801 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:10:16.771525 (+ 724us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:10:16.771527 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:10:16.771958 (+ 431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:10:16.771959 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:10:16.772009 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:10:16.772374 (+ 365us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 21:10:16.772378 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 21:10:16.773703 (+ 1325us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 21:10:16.775314 (+ 1611us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 21:10:16.775315 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 21:10:16.775315 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 21:10:16.775503 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 21:10:16.775665 (+ 162us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 21:10:16.775665 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 21:10:16.775666 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 21:10:16.775729 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 21:10:16.775968 (+ 239us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 21:10:16.775969 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 21:10:16.776008 (+ 39us) client_negotiation.cc:769] Sending connection context
0924 21:10:16.776077 (+ 69us) client_negotiation.cc:240] Negotiation successful
0924 21:10:16.776157 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":61}
I20260924 21:10:16.776242 13890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:10:16.769520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55368 (local address 127.0.98.60:44849)
0924 21:10:16.769577 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0924 21:10:16.769578 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 21:10:16.769586 (+ 8us) server_negotiation.cc:407] Connection header received
0924 21:10:16.769603 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:10:16.769603 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:10:16.769611 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:10:16.769669 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:10:16.770017 (+ 348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:10:16.770710 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:10:16.771609 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:10:16.771884 (+ 275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:10:16.771947 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:10:16.773825 (+ 1878us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 21:10:16.773826 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 21:10:16.773826 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 21:10:16.773833 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 21:10:16.775233 (+ 1400us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:10:16.775571 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:10:16.775571 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:10:16.775572 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:10:16.775616 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:10:16.775794 (+ 178us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:10:16.775795 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:10:16.775795 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:10:16.775899 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 21:10:16.775957 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0924 21:10:16.776110 (+ 153us) server_negotiation.cc:299] Negotiation successful
0924 21:10:16.776169 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260924 21:10:16.777846 13810 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b79e00105bd14bee96da3db6960088c0: Bootstrap starting.
I20260924 21:10:16.778443 13810 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b79e00105bd14bee96da3db6960088c0: Neither blocks nor log segments found. Creating new log.
I20260924 21:10:16.778705 13810 log.cc:824] T 00000000000000000000000000000000 P b79e00105bd14bee96da3db6960088c0: Log is configured to *not* fsync() on all Append() calls
I20260924 21:10:16.778973 13892 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8ccf50cb37604a33a3099ecee0355f0d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 21:10:16.779044 13892 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8ccf50cb37604a33a3099ecee0355f0d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aabb111c8a6b4665b10dd7adc0faa02e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46431 } } peers { permanent_uuid: "b79e00105bd14bee96da3db6960088c0" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39919 } } peers { permanent_uuid: "8ccf50cb37604a33a3099ecee0355f0d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44849 } }
I20260924 21:10:16.779241 13892 leader_election.cc:288] T 00000000000000000000000000000000 P 8ccf50cb37604a33a3099ecee0355f0d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aabb111c8a6b4665b10dd7adc0faa02e (127.0.98.62:46431), b79e00105bd14bee96da3db6960088c0 (127.0.98.61:39919)
I20260924 21:10:16.779505 13810 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b79e00105bd14bee96da3db6960088c0: No bootstrap required, opened a new log
I20260924 21:10:16.779700 13709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8ccf50cb37604a33a3099ecee0355f0d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aabb111c8a6b4665b10dd7adc0faa02e" is_pre_election: true
I20260924 21:10:16.779807 13709 raft_consensus.cc:2454] T 00000000000000000000000000000000 P aabb111c8a6b4665b10dd7adc0faa02e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8ccf50cb37604a33a3099ecee0355f0d in term 0.
I20260924 21:10:16.780041 13835 leader_election.cc:302] T 00000000000000000000000000000000 P 8ccf50cb37604a33a3099ecee0355f0d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8ccf50cb37604a33a3099ecee0355f0d, aabb111c8a6b4665b10dd7adc0faa02e; no voters:
I20260924 21:10:16.780155 13892 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8ccf50cb37604a33a3099ecee0355f0d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 21:10:16.780207 13892 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8ccf50cb37604a33a3099ecee0355f0d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 21:10:16.780229 13892 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8ccf50cb37604a33a3099ecee0355f0d [term 0 FOLLOWER]: Advancing to term 1
I20260924 21:10:16.780963 13892 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8ccf50cb37604a33a3099ecee0355f0d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aabb111c8a6b4665b10dd7adc0faa02e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46431 } } peers { permanent_uuid: "b79e00105bd14bee96da3db6960088c0" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39919 } } peers { permanent_uuid: "8ccf50cb37604a33a3099ecee0355f0d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44849 } }
I20260924 21:10:16.781095 13892 leader_election.cc:288] T 00000000000000000000000000000000 P 8ccf50cb37604a33a3099ecee0355f0d [CANDIDATE]: Term 1 election: Requested vote from peers aabb111c8a6b4665b10dd7adc0faa02e (127.0.98.62:46431), b79e00105bd14bee96da3db6960088c0 (127.0.98.61:39919)
I20260924 21:10:16.781448 13709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8ccf50cb37604a33a3099ecee0355f0d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aabb111c8a6b4665b10dd7adc0faa02e"
I20260924 21:10:16.781512 13709 raft_consensus.cc:3037] T 00000000000000000000000000000000 P aabb111c8a6b4665b10dd7adc0faa02e [term 0 FOLLOWER]: Advancing to term 1
I20260924 21:10:16.781912 13810 raft_consensus.cc:348] T 00000000000000000000000000000000 P b79e00105bd14bee96da3db6960088c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aabb111c8a6b4665b10dd7adc0faa02e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46431 } } peers { permanent_uuid: "b79e00105bd14bee96da3db6960088c0" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39919 } } peers { permanent_uuid: "8ccf50cb37604a33a3099ecee0355f0d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44849 } }
I20260924 21:10:16.782109 13810 raft_consensus.cc:374] T 00000000000000000000000000000000 P b79e00105bd14bee96da3db6960088c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 21:10:16.782189 13810 raft_consensus.cc:729] T 00000000000000000000000000000000 P b79e00105bd14bee96da3db6960088c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b79e00105bd14bee96da3db6960088c0, State: Initialized, Role: FOLLOWER
I20260924 21:10:16.782299 13709 raft_consensus.cc:2454] T 00000000000000000000000000000000 P aabb111c8a6b4665b10dd7adc0faa02e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8ccf50cb37604a33a3099ecee0355f0d in term 1.
I20260924 21:10:16.782413 13810 consensus_queue.cc:260] T 00000000000000000000000000000000 P b79e00105bd14bee96da3db6960088c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aabb111c8a6b4665b10dd7adc0faa02e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46431 } } peers { permanent_uuid: "b79e00105bd14bee96da3db6960088c0" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39919 } } peers { permanent_uuid: "8ccf50cb37604a33a3099ecee0355f0d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44849 } }
I20260924 21:10:16.782500 13835 leader_election.cc:302] T 00000000000000000000000000000000 P 8ccf50cb37604a33a3099ecee0355f0d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8ccf50cb37604a33a3099ecee0355f0d, aabb111c8a6b4665b10dd7adc0faa02e; no voters:
I20260924 21:10:16.782595 13892 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8ccf50cb37604a33a3099ecee0355f0d [term 1 FOLLOWER]: Leader election won for term 1
I20260924 21:10:16.782722 13892 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8ccf50cb37604a33a3099ecee0355f0d [term 1 LEADER]: Becoming Leader. State: Replica: 8ccf50cb37604a33a3099ecee0355f0d, State: Running, Role: LEADER
I20260924 21:10:16.782802 13892 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8ccf50cb37604a33a3099ecee0355f0d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aabb111c8a6b4665b10dd7adc0faa02e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46431 } } peers { permanent_uuid: "b79e00105bd14bee96da3db6960088c0" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39919 } } peers { permanent_uuid: "8ccf50cb37604a33a3099ecee0355f0d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44849 } }
I20260924 21:10:16.782910 13810 sys_catalog.cc:564] T 00000000000000000000000000000000 P b79e00105bd14bee96da3db6960088c0 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 21:10:16.782945 13783 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8ccf50cb37604a33a3099ecee0355f0d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b79e00105bd14bee96da3db6960088c0"
I20260924 21:10:16.783041 13783 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b79e00105bd14bee96da3db6960088c0 [term 0 FOLLOWER]: Advancing to term 1
I20260924 21:10:16.783115 13892 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8ccf50cb37604a33a3099ecee0355f0d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8ccf50cb37604a33a3099ecee0355f0d. Latest consensus state: current_term: 1 leader_uuid: "8ccf50cb37604a33a3099ecee0355f0d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aabb111c8a6b4665b10dd7adc0faa02e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46431 } } peers { permanent_uuid: "b79e00105bd14bee96da3db6960088c0" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39919 } } peers { permanent_uuid: "8ccf50cb37604a33a3099ecee0355f0d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44849 } } }
I20260924 21:10:16.783201 13892 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8ccf50cb37604a33a3099ecee0355f0d [sys.catalog]: This master's current role is: LEADER
I20260924 21:10:16.783331 13784 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8ccf50cb37604a33a3099ecee0355f0d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b79e00105bd14bee96da3db6960088c0" is_pre_election: true
I20260924 21:10:16.783670 13930 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 21:10:16.783816 13930 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 21:10:16.783902 13783 raft_consensus.cc:2454] T 00000000000000000000000000000000 P b79e00105bd14bee96da3db6960088c0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8ccf50cb37604a33a3099ecee0355f0d in term 1.
I20260924 21:10:16.785265 13709 raft_consensus.cc:1261] T 00000000000000000000000000000000 P aabb111c8a6b4665b10dd7adc0faa02e [term 1 FOLLOWER]: Refusing update from remote peer 8ccf50cb37604a33a3099ecee0355f0d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 21:10:16.785588 13783 raft_consensus.cc:1261] T 00000000000000000000000000000000 P b79e00105bd14bee96da3db6960088c0 [term 1 FOLLOWER]: Refusing update from remote peer 8ccf50cb37604a33a3099ecee0355f0d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 21:10:16.785831 13892 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 8ccf50cb37604a33a3099ecee0355f0d [LEADER]: Connected to new peer: Peer: permanent_uuid: "aabb111c8a6b4665b10dd7adc0faa02e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46431 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260924 21:10:16.785889 13763 tablet.cc:2369] T 00000000000000000000000000000000 P b79e00105bd14bee96da3db6960088c0: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 21:10:16.786038 13942 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b79e00105bd14bee96da3db6960088c0: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 21:10:16.786127 13942 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 21:10:16.786828 13913 sys_catalog.cc:455] T 00000000000000000000000000000000 P aabb111c8a6b4665b10dd7adc0faa02e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8ccf50cb37604a33a3099ecee0355f0d. Latest consensus state: current_term: 1 leader_uuid: "8ccf50cb37604a33a3099ecee0355f0d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aabb111c8a6b4665b10dd7adc0faa02e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46431 } } peers { permanent_uuid: "b79e00105bd14bee96da3db6960088c0" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39919 } } peers { permanent_uuid: "8ccf50cb37604a33a3099ecee0355f0d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44849 } } }
I20260924 21:10:16.786895 13913 sys_catalog.cc:458] T 00000000000000000000000000000000 P aabb111c8a6b4665b10dd7adc0faa02e [sys.catalog]: This master's current role is: FOLLOWER
I20260924 21:10:16.787583 13892 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 8ccf50cb37604a33a3099ecee0355f0d [LEADER]: Connected to new peer: Peer: permanent_uuid: "b79e00105bd14bee96da3db6960088c0" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39919 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 21:10:16.790171 13892 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8ccf50cb37604a33a3099ecee0355f0d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8ccf50cb37604a33a3099ecee0355f0d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aabb111c8a6b4665b10dd7adc0faa02e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46431 } } peers { permanent_uuid: "b79e00105bd14bee96da3db6960088c0" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39919 } } peers { permanent_uuid: "8ccf50cb37604a33a3099ecee0355f0d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44849 } } }
I20260924 21:10:16.790249 13892 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8ccf50cb37604a33a3099ecee0355f0d [sys.catalog]: This master's current role is: LEADER
I20260924 21:10:16.790364 13927 sys_catalog.cc:455] T 00000000000000000000000000000000 P b79e00105bd14bee96da3db6960088c0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8ccf50cb37604a33a3099ecee0355f0d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aabb111c8a6b4665b10dd7adc0faa02e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46431 } } peers { permanent_uuid: "b79e00105bd14bee96da3db6960088c0" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39919 } } peers { permanent_uuid: "8ccf50cb37604a33a3099ecee0355f0d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44849 } } }
I20260924 21:10:16.790519 13927 sys_catalog.cc:458] T 00000000000000000000000000000000 P b79e00105bd14bee96da3db6960088c0 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 21:10:16.790666 13927 sys_catalog.cc:455] T 00000000000000000000000000000000 P b79e00105bd14bee96da3db6960088c0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8ccf50cb37604a33a3099ecee0355f0d. Latest consensus state: current_term: 1 leader_uuid: "8ccf50cb37604a33a3099ecee0355f0d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aabb111c8a6b4665b10dd7adc0faa02e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46431 } } peers { permanent_uuid: "b79e00105bd14bee96da3db6960088c0" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39919 } } peers { permanent_uuid: "8ccf50cb37604a33a3099ecee0355f0d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44849 } } }
I20260924 21:10:16.790776 13927 sys_catalog.cc:458] T 00000000000000000000000000000000 P b79e00105bd14bee96da3db6960088c0 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 21:10:16.790820 13931 mvcc.cc:205] Tried to move back new op lower bound from 7333004151950712832 to 7333004151942881280. Current Snapshot: MvccSnapshot[applied={T|T < 7333004151950712832}]
I20260924 21:10:16.790853 13927 sys_catalog.cc:455] T 00000000000000000000000000000000 P b79e00105bd14bee96da3db6960088c0 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8ccf50cb37604a33a3099ecee0355f0d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aabb111c8a6b4665b10dd7adc0faa02e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46431 } } peers { permanent_uuid: "b79e00105bd14bee96da3db6960088c0" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39919 } } peers { permanent_uuid: "8ccf50cb37604a33a3099ecee0355f0d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44849 } } }
I20260924 21:10:16.791015 13927 sys_catalog.cc:458] T 00000000000000000000000000000000 P b79e00105bd14bee96da3db6960088c0 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 21:10:16.790912 13947 mvcc.cc:205] Tried to move back new op lower bound from 7333004151950712832 to 7333004151942881280. Current Snapshot: MvccSnapshot[applied={T|T < 7333004151950712832}]
I20260924 21:10:16.791658 13892 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8ccf50cb37604a33a3099ecee0355f0d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8ccf50cb37604a33a3099ecee0355f0d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aabb111c8a6b4665b10dd7adc0faa02e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46431 } } peers { permanent_uuid: "b79e00105bd14bee96da3db6960088c0" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39919 } } peers { permanent_uuid: "8ccf50cb37604a33a3099ecee0355f0d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44849 } } }
I20260924 21:10:16.791728 13892 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8ccf50cb37604a33a3099ecee0355f0d [sys.catalog]: This master's current role is: LEADER
I20260924 21:10:16.792018 13913 sys_catalog.cc:455] T 00000000000000000000000000000000 P aabb111c8a6b4665b10dd7adc0faa02e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8ccf50cb37604a33a3099ecee0355f0d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aabb111c8a6b4665b10dd7adc0faa02e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46431 } } peers { permanent_uuid: "b79e00105bd14bee96da3db6960088c0" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39919 } } peers { permanent_uuid: "8ccf50cb37604a33a3099ecee0355f0d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44849 } } }
I20260924 21:10:16.792089 13913 sys_catalog.cc:458] T 00000000000000000000000000000000 P aabb111c8a6b4665b10dd7adc0faa02e [sys.catalog]: This master's current role is: FOLLOWER
I20260924 21:10:16.792579 13944 mvcc.cc:205] Tried to move back new op lower bound from 7333004151950712832 to 7333004151942881280. Current Snapshot: MvccSnapshot[applied={T|T < 7333004151950712832}]
I20260924 21:10:16.792986 13930 catalog_manager.cc:1347] Generated new cluster ID: 3c610a74ea434403aa067caebc8d0fb8
I20260924 21:10:16.793030 13930 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 21:10:16.797514 13930 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 21:10:16.798087 13930 catalog_manager.cc:1504] Loading token signing keys...
I20260924 21:10:16.808576 13930 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8ccf50cb37604a33a3099ecee0355f0d: Generated new TSK 0
I20260924 21:10:16.808702 13930 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 21:10:17.766644 13924 catalog_manager.cc:1259] Loaded cluster ID: 3c610a74ea434403aa067caebc8d0fb8
I20260924 21:10:17.766702 13924 catalog_manager.cc:1552] T 00000000000000000000000000000000 P aabb111c8a6b4665b10dd7adc0faa02e: loading cluster ID for follower catalog manager: success
I20260924 21:10:17.767460 13924 catalog_manager.cc:1574] T 00000000000000000000000000000000 P aabb111c8a6b4665b10dd7adc0faa02e: acquiring CA information for follower catalog manager: success
I20260924 21:10:17.767752 13924 catalog_manager.cc:1602] T 00000000000000000000000000000000 P aabb111c8a6b4665b10dd7adc0faa02e: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 21:10:17.786473 13942 catalog_manager.cc:1259] Loaded cluster ID: 3c610a74ea434403aa067caebc8d0fb8
I20260924 21:10:17.786520 13942 catalog_manager.cc:1552] T 00000000000000000000000000000000 P b79e00105bd14bee96da3db6960088c0: loading cluster ID for follower catalog manager: success
I20260924 21:10:17.787171 13942 catalog_manager.cc:1574] T 00000000000000000000000000000000 P b79e00105bd14bee96da3db6960088c0: acquiring CA information for follower catalog manager: success
I20260924 21:10:17.787415 13942 catalog_manager.cc:1602] T 00000000000000000000000000000000 P b79e00105bd14bee96da3db6960088c0: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 21:10:17.860800 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAzNNjS/build/release/bin/kudu with pid 13667
I20260924 21:10:17.865584 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAzNNjS/build/release/bin/kudu with pid 13742
I20260924 21:10:17.869879 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAzNNjS/build/release/bin/kudu with pid 13818
2026-09-24T21:10:17Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (2589 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790284076800039-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 21:10:17 dist-test-slave-c7tk krb5kdc[13963](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 21:10:17 dist-test-slave-c7tk krb5kdc[13963](info): set up 2 sockets
Sep 24 21:10:17 dist-test-slave-c7tk krb5kdc[13963](info): commencing operation
krb5kdc: starting...
W20260924 21:10:18.821421 392 mini_kdc.cc:121] Time spent starting KDC: real 0.933s user 0.000s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790284076800039-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790284076800039-392-0/krb5kdc/test-user.keytab.
Sep 24 21:10:18 dist-test-slave-c7tk krb5kdc[13963](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284218, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T21:10:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T21:10:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790284076800039-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790284076800039-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 21:10:18.918465 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAzNNjS/build/release/bin/kudu
/tmp/dist-test-taskAzNNjS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790284076800039-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790284076800039-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790284076800039-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790284076800039-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43599
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790284076800039-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41223
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43599
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790284076800039-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790284076800039-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790284076800039-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 21:10:18.987727 13976 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 21:10:18.987895 13976 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 21:10:18.987913 13976 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 21:10:18.988845 13976 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 21:10:18.988878 13976 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 21:10:18.988890 13976 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 21:10:18.988900 13976 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 21:10:18.988911 13976 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 21:10:18.990278 13976 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41223
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790284076800039-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790284076800039-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43599
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790284076800039-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43599
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790284076800039-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13976
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790284076800039-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 19:49:06 UTC on ccycloud.quasar-vhqjoc.root.comops.site
build id 4670864
I20260924 21:10:18.990515 13976 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 21:10:18.990727 13976 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 21:10:18.995038 13981 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 21:10:18.995038 13984 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 21:10:18.995055 13982 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 21:10:18.995318 13976 server_base.cc:1034] running on GCE node
I20260924 21:10:18.995522 13976 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 21:10:18.995761 13976 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 21:10:18.996902 13976 hybrid_clock.cc:648] HybridClock initialized: now 1790284218996893 us; error 24 us; skew 500 ppm
Sep 24 21:10:18 dist-test-slave-c7tk krb5kdc[13963](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284218, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 21:10:18.998948 13976 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 21:10:18.999318 13976 webserver.cc:460] Webserver started at http://127.0.98.62:43957/ using document root <none> and password file <none>
I20260924 21:10:18.999511 13976 fs_manager.cc:359] Metadata directory not provided
I20260924 21:10:18.999567 13976 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 21:10:18.999693 13976 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 21:10:19.000571 13976 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790284076800039-392-0/minicluster-data/master-0/data/instance:
uuid: "5dc1bb3129484886bf8f2f8ca73e5311"
format_stamp: "Formatted at 2026-09-24 21:10:18 on dist-test-slave-c7tk"
server_key: "e3c30d60804e0c175d778a7b478205f9"
server_key_iv: "cbbd4de0df774047d157d6217cc1f504"
server_key_version: "encryptionkey@0"
I20260924 21:10:19.000902 13976 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790284076800039-392-0/minicluster-data/master-0/wal/instance:
uuid: "5dc1bb3129484886bf8f2f8ca73e5311"
format_stamp: "Formatted at 2026-09-24 21:10:18 on dist-test-slave-c7tk"
server_key: "e3c30d60804e0c175d778a7b478205f9"
server_key_iv: "cbbd4de0df774047d157d6217cc1f504"
server_key_version: "encryptionkey@0"
I20260924 21:10:19.002087 13976 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 21:10:19.002707 13991 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 21:10:19.002872 13976 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 21:10:19.002956 13976 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790284076800039-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790284076800039-392-0/minicluster-data/master-0/wal
uuid: "5dc1bb3129484886bf8f2f8ca73e5311"
format_stamp: "Formatted at 2026-09-24 21:10:18 on dist-test-slave-c7tk"
server_key: "e3c30d60804e0c175d778a7b478205f9"
server_key_iv: "cbbd4de0df774047d157d6217cc1f504"
server_key_version: "encryptionkey@0"
I20260924 21:10:19.003019 13976 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790284076800039-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790284076800039-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790284076800039-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 21:10:19.011732 13976 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 21:10:19.013193 13976 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 21:10:19.013337 13976 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 21:10:19.016852 13976 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43599
I20260924 21:10:19.016875 14043 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43599 every 8 connection(s)
I20260924 21:10:19.017139 13976 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790284076800039-392-0/minicluster-data/master-0/data/info.pb
I20260924 21:10:19.017958 14044 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 21:10:19.019827 14044 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5dc1bb3129484886bf8f2f8ca73e5311: Bootstrap starting.
I20260924 21:10:19.020337 14044 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5dc1bb3129484886bf8f2f8ca73e5311: Neither blocks nor log segments found. Creating new log.
I20260924 21:10:19.020627 14044 log.cc:824] T 00000000000000000000000000000000 P 5dc1bb3129484886bf8f2f8ca73e5311: Log is configured to *not* fsync() on all Append() calls
I20260924 21:10:19.021247 14044 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5dc1bb3129484886bf8f2f8ca73e5311: No bootstrap required, opened a new log
I20260924 21:10:19.021760 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAzNNjS/build/release/bin/kudu as pid 13976
I20260924 21:10:19.021860 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790284076800039-392-0/minicluster-data/master-0/wal/instance
I20260924 21:10:19.022063 392 external_mini_cluster.cc:1428] Setting key c9e9274aaa64263d775da0516da82fd3
I20260924 21:10:19.023175 14044 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5dc1bb3129484886bf8f2f8ca73e5311 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5dc1bb3129484886bf8f2f8ca73e5311" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43599 } }
I20260924 21:10:19.023304 14044 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5dc1bb3129484886bf8f2f8ca73e5311 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 21:10:19.023365 14044 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5dc1bb3129484886bf8f2f8ca73e5311 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5dc1bb3129484886bf8f2f8ca73e5311, State: Initialized, Role: FOLLOWER
I20260924 21:10:19.023509 14044 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5dc1bb3129484886bf8f2f8ca73e5311 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5dc1bb3129484886bf8f2f8ca73e5311" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43599 } }
I20260924 21:10:19.023594 14044 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5dc1bb3129484886bf8f2f8ca73e5311 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 21:10:19.023631 14044 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5dc1bb3129484886bf8f2f8ca73e5311 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 21:10:19.023671 14044 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5dc1bb3129484886bf8f2f8ca73e5311 [term 0 FOLLOWER]: Advancing to term 1
I20260924 21:10:19.024358 14044 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5dc1bb3129484886bf8f2f8ca73e5311 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5dc1bb3129484886bf8f2f8ca73e5311" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43599 } }
I20260924 21:10:19.024469 14044 leader_election.cc:302] T 00000000000000000000000000000000 P 5dc1bb3129484886bf8f2f8ca73e5311 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5dc1bb3129484886bf8f2f8ca73e5311; no voters:
I20260924 21:10:19.024617 14044 leader_election.cc:288] T 00000000000000000000000000000000 P 5dc1bb3129484886bf8f2f8ca73e5311 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 21:10:19.024637 14049 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5dc1bb3129484886bf8f2f8ca73e5311 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 21:10:19.024729 14049 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5dc1bb3129484886bf8f2f8ca73e5311 [term 1 LEADER]: Becoming Leader. State: Replica: 5dc1bb3129484886bf8f2f8ca73e5311, State: Running, Role: LEADER
I20260924 21:10:19.024837 14044 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5dc1bb3129484886bf8f2f8ca73e5311 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 21:10:19.024847 14049 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5dc1bb3129484886bf8f2f8ca73e5311 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5dc1bb3129484886bf8f2f8ca73e5311" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43599 } }
I20260924 21:10:19.025988 14051 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5dc1bb3129484886bf8f2f8ca73e5311 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5dc1bb3129484886bf8f2f8ca73e5311. Latest consensus state: current_term: 1 leader_uuid: "5dc1bb3129484886bf8f2f8ca73e5311" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5dc1bb3129484886bf8f2f8ca73e5311" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43599 } } }
I20260924 21:10:19.025985 14050 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5dc1bb3129484886bf8f2f8ca73e5311 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5dc1bb3129484886bf8f2f8ca73e5311" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5dc1bb3129484886bf8f2f8ca73e5311" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43599 } } }
I20260924 21:10:19.026135 14051 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5dc1bb3129484886bf8f2f8ca73e5311 [sys.catalog]: This master's current role is: LEADER
I20260924 21:10:19.026206 14050 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5dc1bb3129484886bf8f2f8ca73e5311 [sys.catalog]: This master's current role is: LEADER
W20260924 21:10:19.026489 14064 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5dc1bb3129484886bf8f2f8ca73e5311: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 21:10:19.026528 14064 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
Sep 24 21:10:19 dist-test-slave-c7tk krb5kdc[13963](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284218, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 21:10:19.029920 14066 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 21:10:19.030093 14066 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 21:10:19.031638 14066 catalog_manager.cc:1347] Generated new cluster ID: 35e33eef11634516bc670e3f0c1d8d90
I20260924 21:10:19.031678 14066 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 21:10:19.031972 14048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:10:19.022585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37028 (local address 127.0.98.62:43599)
0924 21:10:19.022715 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0924 21:10:19.022721 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 21:10:19.022740 (+ 19us) server_negotiation.cc:407] Connection header received
0924 21:10:19.023184 (+ 444us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:10:19.023190 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:10:19.023234 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:10:19.023404 (+ 170us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:10:19.024009 (+ 605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:10:19.024956 (+ 947us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:10:19.025876 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:10:19.026414 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:10:19.026512 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:10:19.028673 (+ 2161us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 21:10:19.028677 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 21:10:19.028677 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 21:10:19.028686 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 21:10:19.030390 (+ 1704us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:10:19.030786 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:10:19.030789 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:10:19.030789 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:10:19.030907 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:10:19.031123 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:10:19.031124 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:10:19.031124 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:10:19.031420 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 21:10:19.031554 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0924 21:10:19.031670 (+ 116us) server_negotiation.cc:299] Negotiation successful
0924 21:10:19.031782 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95,"thread_start_us":60,"threads_started":1}
I20260924 21:10:19.032716 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 21:10:19.046592 14066 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 21:10:19.047196 14066 catalog_manager.cc:1504] Loading token signing keys...
I20260924 21:10:19.053197 14066 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5dc1bb3129484886bf8f2f8ca73e5311: Generated new TSK 0
I20260924 21:10:19.053346 14066 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 21:10:19.086222 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAzNNjS/build/release/bin/kudu with pid 13976
2026-09-24T21:10:19Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1214 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790284076800039-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 21:10:19 dist-test-slave-c7tk krb5kdc[14071](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 21:10:19 dist-test-slave-c7tk krb5kdc[14071](info): set up 2 sockets
Sep 24 21:10:19 dist-test-slave-c7tk krb5kdc[14071](info): commencing operation
krb5kdc: starting...
W20260924 21:10:19.989362 392 mini_kdc.cc:121] Time spent starting KDC: real 0.886s user 0.002s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790284076800039-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790284076800039-392-0/krb5kdc/test-user.keytab.
Sep 24 21:10:20 dist-test-slave-c7tk krb5kdc[14071](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284220, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T21:10:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T21:10:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790284076800039-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790284076800039-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 21:10:20.090342 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAzNNjS/build/release/bin/kudu
/tmp/dist-test-taskAzNNjS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790284076800039-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790284076800039-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790284076800039-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790284076800039-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45663
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790284076800039-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35231
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45663
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790284076800039-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790284076800039-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790284076800039-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 21:10:20.159301 14084 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 21:10:20.159462 14084 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 21:10:20.159482 14084 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 21:10:20.160482 14084 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 21:10:20.160516 14084 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 21:10:20.160528 14084 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 21:10:20.160542 14084 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 21:10:20.160553 14084 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 21:10:20.161911 14084 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35231
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790284076800039-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790284076800039-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45663
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790284076800039-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45663
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790284076800039-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14084
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790284076800039-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 19:49:06 UTC on ccycloud.quasar-vhqjoc.root.comops.site
build id 4670864
I20260924 21:10:20.162058 14084 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 21:10:20.162186 14084 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 21:10:20.166018 14092 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 21:10:20.166030 14089 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 21:10:20.166030 14090 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 21:10:20.166175 14084 server_base.cc:1034] running on GCE node
I20260924 21:10:20.166414 14084 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 21:10:20.166647 14084 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 21:10:20.167793 14084 hybrid_clock.cc:648] HybridClock initialized: now 1790284220167785 us; error 34 us; skew 500 ppm
Sep 24 21:10:20 dist-test-slave-c7tk krb5kdc[14071](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284220, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 21:10:20.169840 14084 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 21:10:20.170178 14084 webserver.cc:460] Webserver started at http://127.0.98.62:41717/ using document root <none> and password file <none>
I20260924 21:10:20.170418 14084 fs_manager.cc:359] Metadata directory not provided
I20260924 21:10:20.170482 14084 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 21:10:20.170624 14084 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 21:10:20.171561 14084 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790284076800039-392-0/minicluster-data/master-0/data/instance:
uuid: "4e3cd7d6c2b4441093bfb5a89a0767ae"
format_stamp: "Formatted at 2026-09-24 21:10:20 on dist-test-slave-c7tk"
server_key: "e7de828d96504a2c38339d0e70f025a0"
server_key_iv: "dcbce15300373d651b4546345bfa50e7"
server_key_version: "encryptionkey@0"
I20260924 21:10:20.171873 14084 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790284076800039-392-0/minicluster-data/master-0/wal/instance:
uuid: "4e3cd7d6c2b4441093bfb5a89a0767ae"
format_stamp: "Formatted at 2026-09-24 21:10:20 on dist-test-slave-c7tk"
server_key: "e7de828d96504a2c38339d0e70f025a0"
server_key_iv: "dcbce15300373d651b4546345bfa50e7"
server_key_version: "encryptionkey@0"
I20260924 21:10:20.172924 14084 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 21:10:20.173493 14099 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 21:10:20.173625 14084 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 21:10:20.173707 14084 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790284076800039-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790284076800039-392-0/minicluster-data/master-0/wal
uuid: "4e3cd7d6c2b4441093bfb5a89a0767ae"
format_stamp: "Formatted at 2026-09-24 21:10:20 on dist-test-slave-c7tk"
server_key: "e7de828d96504a2c38339d0e70f025a0"
server_key_iv: "dcbce15300373d651b4546345bfa50e7"
server_key_version: "encryptionkey@0"
I20260924 21:10:20.173774 14084 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790284076800039-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790284076800039-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790284076800039-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 21:10:20.182463 14084 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 21:10:20.184247 14084 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 21:10:20.184382 14084 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 21:10:20.187588 14084 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45663
I20260924 21:10:20.187615 14151 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45663 every 8 connection(s)
I20260924 21:10:20.187882 14084 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790284076800039-392-0/minicluster-data/master-0/data/info.pb
I20260924 21:10:20.188678 14152 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 21:10:20.190554 14152 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4e3cd7d6c2b4441093bfb5a89a0767ae: Bootstrap starting.
I20260924 21:10:20.191073 14152 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4e3cd7d6c2b4441093bfb5a89a0767ae: Neither blocks nor log segments found. Creating new log.
I20260924 21:10:20.191339 14152 log.cc:824] T 00000000000000000000000000000000 P 4e3cd7d6c2b4441093bfb5a89a0767ae: Log is configured to *not* fsync() on all Append() calls
I20260924 21:10:20.191920 14152 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4e3cd7d6c2b4441093bfb5a89a0767ae: No bootstrap required, opened a new log
I20260924 21:10:20.193236 14152 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4e3cd7d6c2b4441093bfb5a89a0767ae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e3cd7d6c2b4441093bfb5a89a0767ae" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45663 } }
I20260924 21:10:20.193351 14152 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4e3cd7d6c2b4441093bfb5a89a0767ae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 21:10:20.193406 14152 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4e3cd7d6c2b4441093bfb5a89a0767ae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e3cd7d6c2b4441093bfb5a89a0767ae, State: Initialized, Role: FOLLOWER
I20260924 21:10:20.193560 14152 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4e3cd7d6c2b4441093bfb5a89a0767ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e3cd7d6c2b4441093bfb5a89a0767ae" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45663 } }
I20260924 21:10:20.193616 14152 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4e3cd7d6c2b4441093bfb5a89a0767ae [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 21:10:20.193670 14152 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4e3cd7d6c2b4441093bfb5a89a0767ae [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 21:10:20.193714 14152 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4e3cd7d6c2b4441093bfb5a89a0767ae [term 0 FOLLOWER]: Advancing to term 1
I20260924 21:10:20.194118 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAzNNjS/build/release/bin/kudu as pid 14084
I20260924 21:10:20.194208 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790284076800039-392-0/minicluster-data/master-0/wal/instance
I20260924 21:10:20.194398 392 external_mini_cluster.cc:1428] Setting key cdf4a8a7bc7a60061219b7245ada0f8a
I20260924 21:10:20.194516 14152 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4e3cd7d6c2b4441093bfb5a89a0767ae [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e3cd7d6c2b4441093bfb5a89a0767ae" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45663 } }
I20260924 21:10:20.194620 14152 leader_election.cc:302] T 00000000000000000000000000000000 P 4e3cd7d6c2b4441093bfb5a89a0767ae [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4e3cd7d6c2b4441093bfb5a89a0767ae; no voters:
I20260924 21:10:20.194823 14152 leader_election.cc:288] T 00000000000000000000000000000000 P 4e3cd7d6c2b4441093bfb5a89a0767ae [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 21:10:20.194852 14156 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4e3cd7d6c2b4441093bfb5a89a0767ae [term 1 FOLLOWER]: Leader election won for term 1
I20260924 21:10:20.194983 14156 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4e3cd7d6c2b4441093bfb5a89a0767ae [term 1 LEADER]: Becoming Leader. State: Replica: 4e3cd7d6c2b4441093bfb5a89a0767ae, State: Running, Role: LEADER
I20260924 21:10:20.195024 14152 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4e3cd7d6c2b4441093bfb5a89a0767ae [sys.catalog]: configured and running, proceeding with master startup.
I20260924 21:10:20.195111 14156 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4e3cd7d6c2b4441093bfb5a89a0767ae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e3cd7d6c2b4441093bfb5a89a0767ae" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45663 } }
I20260924 21:10:20.195442 14159 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4e3cd7d6c2b4441093bfb5a89a0767ae [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4e3cd7d6c2b4441093bfb5a89a0767ae" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e3cd7d6c2b4441093bfb5a89a0767ae" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45663 } } }
I20260924 21:10:20.195538 14159 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4e3cd7d6c2b4441093bfb5a89a0767ae [sys.catalog]: This master's current role is: LEADER
I20260924 21:10:20.195632 14158 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4e3cd7d6c2b4441093bfb5a89a0767ae [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4e3cd7d6c2b4441093bfb5a89a0767ae. Latest consensus state: current_term: 1 leader_uuid: "4e3cd7d6c2b4441093bfb5a89a0767ae" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e3cd7d6c2b4441093bfb5a89a0767ae" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45663 } } }
I20260924 21:10:20.195832 14158 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4e3cd7d6c2b4441093bfb5a89a0767ae [sys.catalog]: This master's current role is: LEADER
I20260924 21:10:20.195966 14165 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 21:10:20.196487 14165 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 21:10:20.197889 14165 catalog_manager.cc:1347] Generated new cluster ID: 29f9a19ac9f14577a5de672c45a0000e
I20260924 21:10:20.198151 14165 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 24 21:10:20 dist-test-slave-c7tk krb5kdc[14071](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284220, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 21:10:20.203380 14157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:10:20.194737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47416 (local address 127.0.98.62:45663)
0924 21:10:20.195038 (+ 301us) server_negotiation.cc:206] Beginning negotiation
0924 21:10:20.195047 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 21:10:20.195074 (+ 27us) server_negotiation.cc:407] Connection header received
0924 21:10:20.195163 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:10:20.195169 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:10:20.195220 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:10:20.195359 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:10:20.196074 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:10:20.197185 (+ 1111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:10:20.198066 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:10:20.198484 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:10:20.198596 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:10:20.200529 (+ 1933us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 21:10:20.200532 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 21:10:20.200533 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 21:10:20.200551 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0924 21:10:20.202104 (+ 1553us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:10:20.202432 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:10:20.202435 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:10:20.202435 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:10:20.202515 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:10:20.202694 (+ 179us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:10:20.202695 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:10:20.202695 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:10:20.202900 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 21:10:20.203043 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0924 21:10:20.203098 (+ 55us) server_negotiation.cc:299] Negotiation successful
0924 21:10:20.203211 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":261,"thread_start_us":73,"threads_started":1}
I20260924 21:10:20.223294 14165 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 21:10:20.223843 14165 catalog_manager.cc:1504] Loading token signing keys...
I20260924 21:10:20.229244 14165 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4e3cd7d6c2b4441093bfb5a89a0767ae: Generated new TSK 0
I20260924 21:10:20.229390 14165 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790284076800039-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790284076800039-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260924 21:10:20.256075 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAzNNjS/build/release/bin/kudu
/tmp/dist-test-taskAzNNjS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790284076800039-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790284076800039-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790284076800039-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790284076800039-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45663
--keytab_file=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790284076800039-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35231
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790284076800039-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790284076800039-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790284076800039-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 21:10:20.332026 14178 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 21:10:20.332227 14178 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 21:10:20.332249 14178 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 21:10:20.333387 14178 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 21:10:20.333429 14178 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 21:10:20.333477 14178 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 21:10:20.334942 14178 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35231
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790284076800039-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790284076800039-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790284076800039-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790284076800039-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45663
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14178
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790284076800039-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 19:49:06 UTC on ccycloud.quasar-vhqjoc.root.comops.site
build id 4670864
I20260924 21:10:20.335170 14178 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 21:10:20.335393 14178 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 21:10:20.340062 14184 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 21:10:20.340075 14186 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 21:10:20.340178 14178 server_base.cc:1034] running on GCE node
W20260924 21:10:20.340106 14183 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 21:10:20.340502 14178 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 21:10:20.340734 14178 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 21:10:20.341867 14178 hybrid_clock.cc:648] HybridClock initialized: now 1790284220341850 us; error 32 us; skew 500 ppm
Sep 24 21:10:20 dist-test-slave-c7tk krb5kdc[14071](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284220, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 21:10:20.344161 14178 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 21:10:20.344547 14178 webserver.cc:460] Webserver started at http://127.0.98.1:38341/ using document root <none> and password file <none>
I20260924 21:10:20.344761 14178 fs_manager.cc:359] Metadata directory not provided
I20260924 21:10:20.344827 14178 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 21:10:20.344990 14178 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 21:10:20.346011 14178 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790284076800039-392-0/minicluster-data/ts-0/data/instance:
uuid: "98dbb90780044a349f025e385e51e5c7"
format_stamp: "Formatted at 2026-09-24 21:10:20 on dist-test-slave-c7tk"
server_key: "ad8242e15fbb6b838111c50417b89741"
server_key_iv: "a745b8c1045ad53db9fd3d021a833161"
server_key_version: "encryptionkey@0"
I20260924 21:10:20.346371 14178 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790284076800039-392-0/minicluster-data/ts-0/wal/instance:
uuid: "98dbb90780044a349f025e385e51e5c7"
format_stamp: "Formatted at 2026-09-24 21:10:20 on dist-test-slave-c7tk"
server_key: "ad8242e15fbb6b838111c50417b89741"
server_key_iv: "a745b8c1045ad53db9fd3d021a833161"
server_key_version: "encryptionkey@0"
I20260924 21:10:20.347635 14178 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 21:10:20.348321 14193 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 21:10:20.348477 14178 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 21:10:20.348557 14178 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790284076800039-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790284076800039-392-0/minicluster-data/ts-0/wal
uuid: "98dbb90780044a349f025e385e51e5c7"
format_stamp: "Formatted at 2026-09-24 21:10:20 on dist-test-slave-c7tk"
server_key: "ad8242e15fbb6b838111c50417b89741"
server_key_iv: "a745b8c1045ad53db9fd3d021a833161"
server_key_version: "encryptionkey@0"
I20260924 21:10:20.348636 14178 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790284076800039-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790284076800039-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790284076800039-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 21:10:20.356935 14178 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 21:10:20.358629 14178 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 21:10:20.358777 14178 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 21:10:20.358996 14178 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 21:10:20.359266 14178 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 21:10:20.359318 14178 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 21:10:20.359350 14178 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 21:10:20.359376 14178 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 21:10:20.364882 14178 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:44771
I20260924 21:10:20.364913 14306 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:44771 every 8 connection(s)
I20260924 21:10:20.365168 14178 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790284076800039-392-0/minicluster-data/ts-0/data/info.pb
I20260924 21:10:20.369980 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAzNNjS/build/release/bin/kudu as pid 14178
I20260924 21:10:20.370057 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790284076800039-392-0/minicluster-data/ts-0/wal/instance
I20260924 21:10:20.370260 392 external_mini_cluster.cc:1428] Setting key 87a868cb759141a9ab3bef2e3d92bd6b
Sep 24 21:10:20 dist-test-slave-c7tk krb5kdc[14071](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284220, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 21:10:20.373617 14157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:10:20.365856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:34179 (local address 127.0.98.62:45663)
0924 21:10:20.365898 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0924 21:10:20.365905 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 21:10:20.366055 (+ 150us) server_negotiation.cc:407] Connection header received
0924 21:10:20.366460 (+ 405us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:10:20.366461 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:10:20.366474 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:10:20.366536 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:10:20.367286 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:10:20.367734 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:10:20.368575 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:10:20.368752 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:10:20.368813 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:10:20.370893 (+ 2080us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 21:10:20.370895 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 21:10:20.370895 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 21:10:20.370904 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 21:10:20.372288 (+ 1384us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:10:20.372740 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:10:20.372741 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:10:20.372742 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:10:20.372796 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:10:20.373022 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:10:20.373023 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:10:20.373023 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:10:20.373147 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 21:10:20.373233 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0924 21:10:20.373499 (+ 266us) server_negotiation.cc:299] Negotiation successful
0924 21:10:20.373559 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260924 21:10:20.373862 14309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:10:20.365782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45663 (local address 127.0.98.1:34179)
0924 21:10:20.365955 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0924 21:10:20.365990 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0924 21:10:20.366381 (+ 391us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 21:10:20.366579 (+ 198us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 21:10:20.366582 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 21:10:20.367059 (+ 477us) client_negotiation.cc:189] Negotiated authn=SASL
0924 21:10:20.367215 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:10:20.367219 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:10:20.367808 (+ 589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:10:20.367809 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:10:20.368502 (+ 693us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:10:20.368504 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:10:20.368819 (+ 315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:10:20.368820 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:10:20.368909 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:10:20.369293 (+ 384us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 21:10:20.369300 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0924 21:10:20.370781 (+ 1481us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 21:10:20.372420 (+ 1639us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 21:10:20.372427 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 21:10:20.372428 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 21:10:20.372648 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 21:10:20.372856 (+ 208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 21:10:20.372857 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 21:10:20.372857 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 21:10:20.372955 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 21:10:20.373224 (+ 269us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 21:10:20.373230 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 21:10:20.373370 (+ 140us) client_negotiation.cc:769] Sending connection context
0924 21:10:20.373469 (+ 99us) client_negotiation.cc:240] Negotiation successful
0924 21:10:20.373615 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":69,"threads_started":1}
I20260924 21:10:20.374261 14307 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45663
I20260924 21:10:20.374342 14307 heartbeater.cc:464] Registering TS with master...
I20260924 21:10:20.374531 14307 heartbeater.cc:511] Master 127.0.98.62:45663 requested a full tablet report, sending...
I20260924 21:10:20.375006 14116 ts_manager.cc:194] Registered new tserver with Master: 98dbb90780044a349f025e385e51e5c7 (127.0.98.1:44771)
I20260924 21:10:20.375573 14116 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:34179
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790284076800039-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790284076800039-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260924 21:10:20.393023 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAzNNjS/build/release/bin/kudu
/tmp/dist-test-taskAzNNjS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790284076800039-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790284076800039-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790284076800039-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790284076800039-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45663
--keytab_file=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790284076800039-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35231
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790284076800039-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790284076800039-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790284076800039-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 21:10:20.467603 14312 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 21:10:20.467764 14312 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 21:10:20.467784 14312 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 21:10:20.468711 14312 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 21:10:20.468744 14312 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 21:10:20.468767 14312 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 21:10:20.469942 14312 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35231
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790284076800039-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790284076800039-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790284076800039-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790284076800039-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45663
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14312
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790284076800039-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 19:49:06 UTC on ccycloud.quasar-vhqjoc.root.comops.site
build id 4670864
I20260924 21:10:20.470098 14312 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 21:10:20.470239 14312 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 21:10:20.474781 14318 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 21:10:20.474773 14320 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 21:10:20.474822 14317 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 21:10:20.474957 14312 server_base.cc:1034] running on GCE node
I20260924 21:10:20.475127 14312 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 21:10:20.475410 14312 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 21:10:20.476572 14312 hybrid_clock.cc:648] HybridClock initialized: now 1790284220476560 us; error 28 us; skew 500 ppm
Sep 24 21:10:20 dist-test-slave-c7tk krb5kdc[14071](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284220, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 21:10:20.478755 14312 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 21:10:20.479104 14312 webserver.cc:460] Webserver started at http://127.0.98.2:46117/ using document root <none> and password file <none>
I20260924 21:10:20.479301 14312 fs_manager.cc:359] Metadata directory not provided
I20260924 21:10:20.479363 14312 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 21:10:20.479493 14312 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 21:10:20.480413 14312 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790284076800039-392-0/minicluster-data/ts-1/data/instance:
uuid: "6125c714aa5640fba1a78fb9c555e3d2"
format_stamp: "Formatted at 2026-09-24 21:10:20 on dist-test-slave-c7tk"
server_key: "285202fe52dbb23bcbcc83d50a1c0497"
server_key_iv: "dfc07fb97f92866d307feec1d056a08e"
server_key_version: "encryptionkey@0"
I20260924 21:10:20.480749 14312 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790284076800039-392-0/minicluster-data/ts-1/wal/instance:
uuid: "6125c714aa5640fba1a78fb9c555e3d2"
format_stamp: "Formatted at 2026-09-24 21:10:20 on dist-test-slave-c7tk"
server_key: "285202fe52dbb23bcbcc83d50a1c0497"
server_key_iv: "dfc07fb97f92866d307feec1d056a08e"
server_key_version: "encryptionkey@0"
I20260924 21:10:20.482025 14312 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 21:10:20.482678 14327 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 21:10:20.482854 14312 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 21:10:20.482951 14312 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790284076800039-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790284076800039-392-0/minicluster-data/ts-1/wal
uuid: "6125c714aa5640fba1a78fb9c555e3d2"
format_stamp: "Formatted at 2026-09-24 21:10:20 on dist-test-slave-c7tk"
server_key: "285202fe52dbb23bcbcc83d50a1c0497"
server_key_iv: "dfc07fb97f92866d307feec1d056a08e"
server_key_version: "encryptionkey@0"
I20260924 21:10:20.483039 14312 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790284076800039-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790284076800039-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790284076800039-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 21:10:20.500077 14312 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 21:10:20.502000 14312 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 21:10:20.502132 14312 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 21:10:20.502336 14312 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 21:10:20.502581 14312 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 21:10:20.502629 14312 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 21:10:20.502660 14312 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 21:10:20.502682 14312 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 21:10:20.507797 14312 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:41909
I20260924 21:10:20.507836 14440 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:41909 every 8 connection(s)
I20260924 21:10:20.508124 14312 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790284076800039-392-0/minicluster-data/ts-1/data/info.pb
Sep 24 21:10:20 dist-test-slave-c7tk krb5kdc[14071](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284220, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 21:10:20.516244 14157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:10:20.508756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:36789 (local address 127.0.98.62:45663)
0924 21:10:20.508796 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0924 21:10:20.508798 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 21:10:20.508969 (+ 171us) server_negotiation.cc:407] Connection header received
0924 21:10:20.509495 (+ 526us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:10:20.509497 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:10:20.509509 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:10:20.509561 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:10:20.510499 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:10:20.510996 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:10:20.511939 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:10:20.512123 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:10:20.512200 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:10:20.514198 (+ 1998us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 21:10:20.514199 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 21:10:20.514200 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 21:10:20.514206 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 21:10:20.515272 (+ 1066us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:10:20.515587 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:10:20.515587 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:10:20.515588 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:10:20.515630 (+ 42us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:10:20.515804 (+ 174us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:10:20.515804 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:10:20.515805 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:10:20.515885 (+ 80us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 21:10:20.515926 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0924 21:10:20.516133 (+ 207us) server_negotiation.cc:299] Negotiation successful
0924 21:10:20.516185 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260924 21:10:20.516378 14443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:10:20.508722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45663 (local address 127.0.98.2:36789)
0924 21:10:20.508854 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0924 21:10:20.508893 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0924 21:10:20.509385 (+ 492us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 21:10:20.509622 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 21:10:20.509626 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 21:10:20.510211 (+ 585us) client_negotiation.cc:189] Negotiated authn=SASL
0924 21:10:20.510390 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:10:20.510394 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:10:20.511088 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:10:20.511089 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:10:20.511852 (+ 763us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:10:20.511854 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:10:20.512211 (+ 357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:10:20.512212 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:10:20.512323 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:10:20.512787 (+ 464us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 21:10:20.512795 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 21:10:20.514119 (+ 1324us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 21:10:20.515342 (+ 1223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 21:10:20.515347 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 21:10:20.515348 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 21:10:20.515526 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 21:10:20.515679 (+ 153us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 21:10:20.515679 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 21:10:20.515679 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 21:10:20.515757 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 21:10:20.515932 (+ 175us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 21:10:20.515935 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 21:10:20.516023 (+ 88us) client_negotiation.cc:769] Sending connection context
0924 21:10:20.516109 (+ 86us) client_negotiation.cc:240] Negotiation successful
0924 21:10:20.516205 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":60,"threads_started":1}
I20260924 21:10:20.516845 14441 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45663
I20260924 21:10:20.516950 14441 heartbeater.cc:464] Registering TS with master...
I20260924 21:10:20.516954 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAzNNjS/build/release/bin/kudu as pid 14312
I20260924 21:10:20.517036 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790284076800039-392-0/minicluster-data/ts-1/wal/instance
I20260924 21:10:20.517112 14441 heartbeater.cc:511] Master 127.0.98.62:45663 requested a full tablet report, sending...
I20260924 21:10:20.517232 392 external_mini_cluster.cc:1428] Setting key 027828d478f19811e1e6a9ff20362ebd
I20260924 21:10:20.517477 14116 ts_manager.cc:194] Registered new tserver with Master: 6125c714aa5640fba1a78fb9c555e3d2 (127.0.98.2:41909)
I20260924 21:10:20.518066 14116 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:36789
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790284076800039-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790284076800039-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260924 21:10:20.540122 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAzNNjS/build/release/bin/kudu
/tmp/dist-test-taskAzNNjS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790284076800039-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790284076800039-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790284076800039-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790284076800039-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45663
--keytab_file=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790284076800039-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35231
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790284076800039-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790284076800039-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790284076800039-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 21:10:20.614972 14446 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 21:10:20.615171 14446 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 21:10:20.615196 14446 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 21:10:20.616253 14446 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 21:10:20.616297 14446 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 21:10:20.616330 14446 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 21:10:20.617484 14446 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35231
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790284076800039-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790284076800039-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790284076800039-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790284076800039-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45663
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14446
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790284076800039-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 19:49:06 UTC on ccycloud.quasar-vhqjoc.root.comops.site
build id 4670864
I20260924 21:10:20.617646 14446 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 21:10:20.617877 14446 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 21:10:20.622695 14454 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 21:10:20.622735 14446 server_base.cc:1034] running on GCE node
W20260924 21:10:20.622676 14452 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 21:10:20.622649 14451 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 21:10:20.623078 14446 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 21:10:20.623312 14446 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 21:10:20.624480 14446 hybrid_clock.cc:648] HybridClock initialized: now 1790284220624470 us; error 26 us; skew 500 ppm
Sep 24 21:10:20 dist-test-slave-c7tk krb5kdc[14071](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284220, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 21:10:20.626860 14446 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 21:10:20.627173 14446 webserver.cc:460] Webserver started at http://127.0.98.3:40127/ using document root <none> and password file <none>
I20260924 21:10:20.627316 14446 fs_manager.cc:359] Metadata directory not provided
I20260924 21:10:20.627358 14446 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 21:10:20.627440 14446 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 21:10:20.628225 14446 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790284076800039-392-0/minicluster-data/ts-2/data/instance:
uuid: "985271ad73854c8eb773b3da78a409dc"
format_stamp: "Formatted at 2026-09-24 21:10:20 on dist-test-slave-c7tk"
server_key: "08f89c81555537cd2f66dbbeaf5bf73e"
server_key_iv: "e72cffdb59c19b9a104557b64137e840"
server_key_version: "encryptionkey@0"
I20260924 21:10:20.628477 14446 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790284076800039-392-0/minicluster-data/ts-2/wal/instance:
uuid: "985271ad73854c8eb773b3da78a409dc"
format_stamp: "Formatted at 2026-09-24 21:10:20 on dist-test-slave-c7tk"
server_key: "08f89c81555537cd2f66dbbeaf5bf73e"
server_key_iv: "e72cffdb59c19b9a104557b64137e840"
server_key_version: "encryptionkey@0"
I20260924 21:10:20.629498 14446 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 21:10:20.630084 14461 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 21:10:20.630218 14446 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 21:10:20.630324 14446 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790284076800039-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790284076800039-392-0/minicluster-data/ts-2/wal
uuid: "985271ad73854c8eb773b3da78a409dc"
format_stamp: "Formatted at 2026-09-24 21:10:20 on dist-test-slave-c7tk"
server_key: "08f89c81555537cd2f66dbbeaf5bf73e"
server_key_iv: "e72cffdb59c19b9a104557b64137e840"
server_key_version: "encryptionkey@0"
I20260924 21:10:20.630373 14446 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790284076800039-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790284076800039-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790284076800039-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 21:10:20.637859 14446 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 21:10:20.639780 14446 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 21:10:20.639890 14446 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 21:10:20.640056 14446 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 21:10:20.640298 14446 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 21:10:20.640347 14446 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 21:10:20.640369 14446 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 21:10:20.640384 14446 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 21:10:20.646436 14446 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:46821
I20260924 21:10:20.646493 14574 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:46821 every 8 connection(s)
I20260924 21:10:20.646690 14446 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790284076800039-392-0/minicluster-data/ts-2/data/info.pb
Sep 24 21:10:20 dist-test-slave-c7tk krb5kdc[14071](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284220, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 21:10:20.654035 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAzNNjS/build/release/bin/kudu as pid 14446
I20260924 21:10:20.654140 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790284076800039-392-0/minicluster-data/ts-2/wal/instance
I20260924 21:10:20.654348 392 external_mini_cluster.cc:1428] Setting key 22d2b6ab7f7f1de7054cf1948571dd14
I20260924 21:10:20.655151 14157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:10:20.647227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:51011 (local address 127.0.98.62:45663)
0924 21:10:20.647265 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0924 21:10:20.647267 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 21:10:20.647454 (+ 187us) server_negotiation.cc:407] Connection header received
0924 21:10:20.647897 (+ 443us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:10:20.647898 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:10:20.647911 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:10:20.647966 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:10:20.648774 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:10:20.649206 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:10:20.650175 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:10:20.650415 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:10:20.650520 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:10:20.652546 (+ 2026us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 21:10:20.652548 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 21:10:20.652548 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 21:10:20.652559 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 21:10:20.653717 (+ 1158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:10:20.654158 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:10:20.654160 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:10:20.654160 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:10:20.654238 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:10:20.654530 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:10:20.654532 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:10:20.654532 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:10:20.654655 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 21:10:20.654713 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0924 21:10:20.655016 (+ 303us) server_negotiation.cc:299] Negotiation successful
0924 21:10:20.655088 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260924 21:10:20.655315 14577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:10:20.647196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45663 (local address 127.0.98.3:51011)
0924 21:10:20.647317 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0924 21:10:20.647356 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0924 21:10:20.647778 (+ 422us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 21:10:20.648023 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 21:10:20.648027 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 21:10:20.648512 (+ 485us) client_negotiation.cc:189] Negotiated authn=SASL
0924 21:10:20.648667 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:10:20.648670 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:10:20.649288 (+ 618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:10:20.649289 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:10:20.650092 (+ 803us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:10:20.650094 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:10:20.650526 (+ 432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:10:20.650527 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:10:20.650645 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:10:20.651104 (+ 459us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 21:10:20.651111 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0924 21:10:20.652449 (+ 1338us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 21:10:20.653794 (+ 1345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 21:10:20.653839 (+ 45us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 21:10:20.653841 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 21:10:20.654062 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 21:10:20.654340 (+ 278us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 21:10:20.654341 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 21:10:20.654341 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 21:10:20.654464 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 21:10:20.654753 (+ 289us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 21:10:20.654759 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 21:10:20.654894 (+ 135us) client_negotiation.cc:769] Sending connection context
0924 21:10:20.654979 (+ 85us) client_negotiation.cc:240] Negotiation successful
0924 21:10:20.655118 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":90,"thread_start_us":64,"threads_started":1}
I20260924 21:10:20.655740 14575 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45663
I20260924 21:10:20.655824 14575 heartbeater.cc:464] Registering TS with master...
I20260924 21:10:20.655965 14575 heartbeater.cc:511] Master 127.0.98.62:45663 requested a full tablet report, sending...
I20260924 21:10:20.656291 14116 ts_manager.cc:194] Registered new tserver with Master: 985271ad73854c8eb773b3da78a409dc (127.0.98.3:46821)
I20260924 21:10:20.656579 14116 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:51011
I20260924 21:10:20.665946 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 21:10:20.672849 14157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:10:20.667214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47424 (local address 127.0.98.62:45663)
0924 21:10:20.667275 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0924 21:10:20.667277 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 21:10:20.667285 (+ 8us) server_negotiation.cc:407] Connection header received
0924 21:10:20.667372 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:10:20.667373 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:10:20.667383 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:10:20.667427 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:10:20.667913 (+ 486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:10:20.668354 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:10:20.669049 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:10:20.669248 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:10:20.669328 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:10:20.670464 (+ 1136us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 21:10:20.670466 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 21:10:20.670466 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 21:10:20.670473 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 21:10:20.671673 (+ 1200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:10:20.672048 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:10:20.672049 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:10:20.672049 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:10:20.672104 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:10:20.672312 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:10:20.672312 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:10:20.672313 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:10:20.672469 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 21:10:20.672575 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0924 21:10:20.672680 (+ 105us) server_negotiation.cc:299] Negotiation successful
0924 21:10:20.672752 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260924 21:10:20.674459 14116 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47424:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 21:10:20.676056 14116 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 21:10:20.682276 14585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:10:20.678070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44771 (local address 127.0.0.1:57138)
0924 21:10:20.678561 (+ 491us) negotiation.cc:107] Waiting for socket to connect
0924 21:10:20.678569 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0924 21:10:20.678610 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 21:10:20.678922 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 21:10:20.678923 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 21:10:20.678924 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 21:10:20.678970 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:10:20.678972 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:10:20.680225 (+ 1253us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:10:20.680226 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:10:20.681104 (+ 878us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:10:20.681116 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:10:20.682016 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:10:20.682016 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:10:20.682071 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:10:20.682081 (+ 10us) client_negotiation.cc:769] Sending connection context
0924 21:10:20.682149 (+ 68us) client_negotiation.cc:240] Negotiation successful
0924 21:10:20.682205 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":475,"thread_start_us":47,"threads_started":1}
I20260924 21:10:20.682304 14586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:10:20.678175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57138 (local address 127.0.98.1:44771)
0924 21:10:20.678603 (+ 428us) server_negotiation.cc:206] Beginning negotiation
0924 21:10:20.678607 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 21:10:20.678617 (+ 10us) server_negotiation.cc:407] Connection header received
0924 21:10:20.678658 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:10:20.678660 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:10:20.678674 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:10:20.678743 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 21:10:20.679106 (+ 363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:10:20.680132 (+ 1026us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:10:20.681199 (+ 1067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:10:20.681941 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:10:20.682017 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:10:20.682070 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0924 21:10:20.682183 (+ 113us) server_negotiation.cc:299] Negotiation successful
0924 21:10:20.682220 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":392,"thread_start_us":73,"threads_started":1}
I20260924 21:10:20.682430 14584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:10:20.678097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41909 (local address 127.0.0.1:55516)
0924 21:10:20.678691 (+ 594us) negotiation.cc:107] Waiting for socket to connect
0924 21:10:20.678698 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0924 21:10:20.678738 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 21:10:20.679002 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 21:10:20.679003 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 21:10:20.679006 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 21:10:20.679041 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:10:20.679042 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:10:20.679973 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:10:20.679974 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:10:20.681286 (+ 1312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:10:20.681289 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:10:20.682210 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:10:20.682210 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:10:20.682262 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:10:20.682265 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 21:10:20.682314 (+ 49us) client_negotiation.cc:240] Negotiation successful
0924 21:10:20.682354 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":585,"mutex_wait_us":28,"thread_start_us":63,"threads_started":1}
I20260924 21:10:20.682641 14588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:10:20.678292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55516 (local address 127.0.98.2:41909)
0924 21:10:20.678496 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0924 21:10:20.678500 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 21:10:20.678796 (+ 296us) server_negotiation.cc:407] Connection header received
0924 21:10:20.678841 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:10:20.678843 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:10:20.678859 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:10:20.678926 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 21:10:20.679110 (+ 184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:10:20.679876 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:10:20.681381 (+ 1505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:10:20.682098 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:10:20.682203 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:10:20.682253 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0924 21:10:20.682495 (+ 242us) server_negotiation.cc:299] Negotiation successful
0924 21:10:20.682538 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":178,"thread_start_us":102,"threads_started":1}
I20260924 21:10:20.683177 14241 tablet_service.cc:1467] Processing CreateTablet for tablet 21d54bdd881a4d4888e1a1acc512bd14 (DEFAULT_TABLE table=test-table [id=7dc021814272415dbe159263a5ae9e75]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 21:10:20.683537 14241 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 21d54bdd881a4d4888e1a1acc512bd14. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 21:10:20.683606 14583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:10:20.678061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:46821 (local address 127.0.0.1:37544)
0924 21:10:20.678227 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0924 21:10:20.678265 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0924 21:10:20.678365 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 21:10:20.678711 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 21:10:20.678714 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 21:10:20.678716 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 21:10:20.678783 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:10:20.678785 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:10:20.680029 (+ 1244us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:10:20.680029 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:10:20.681294 (+ 1265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:10:20.681296 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:10:20.683392 (+ 2096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:10:20.683393 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:10:20.683451 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:10:20.683454 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 21:10:20.683506 (+ 52us) client_negotiation.cc:240] Negotiation successful
0924 21:10:20.683541 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"mutex_wait_us":55,"thread_start_us":64,"threads_started":1}
I20260924 21:10:20.683612 14587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:10:20.678278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37544 (local address 127.0.98.3:46821)
0924 21:10:20.678513 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0924 21:10:20.678518 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 21:10:20.678528 (+ 10us) server_negotiation.cc:407] Connection header received
0924 21:10:20.678583 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:10:20.678585 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:10:20.678600 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:10:20.678683 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 21:10:20.678863 (+ 180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:10:20.679954 (+ 1091us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:10:20.682285 (+ 2331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:10:20.683295 (+ 1010us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:10:20.683387 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:10:20.683435 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0924 21:10:20.683517 (+ 82us) server_negotiation.cc:299] Negotiation successful
0924 21:10:20.683547 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":200,"thread_start_us":105,"threads_started":1}
I20260924 21:10:20.684154 14509 tablet_service.cc:1467] Processing CreateTablet for tablet 21d54bdd881a4d4888e1a1acc512bd14 (DEFAULT_TABLE table=test-table [id=7dc021814272415dbe159263a5ae9e75]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 21:10:20.684420 14509 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 21d54bdd881a4d4888e1a1acc512bd14. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 21:10:20.686554 14589 tablet_bootstrap.cc:492] T 21d54bdd881a4d4888e1a1acc512bd14 P 98dbb90780044a349f025e385e51e5c7: Bootstrap starting.
I20260924 21:10:20.687263 14589 tablet_bootstrap.cc:654] T 21d54bdd881a4d4888e1a1acc512bd14 P 98dbb90780044a349f025e385e51e5c7: Neither blocks nor log segments found. Creating new log.
I20260924 21:10:20.687186 14375 tablet_service.cc:1467] Processing CreateTablet for tablet 21d54bdd881a4d4888e1a1acc512bd14 (DEFAULT_TABLE table=test-table [id=7dc021814272415dbe159263a5ae9e75]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 21:10:20.687489 14375 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 21d54bdd881a4d4888e1a1acc512bd14. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 21:10:20.687825 14589 log.cc:824] T 21d54bdd881a4d4888e1a1acc512bd14 P 98dbb90780044a349f025e385e51e5c7: Log is configured to *not* fsync() on all Append() calls
I20260924 21:10:20.688573 14589 tablet_bootstrap.cc:492] T 21d54bdd881a4d4888e1a1acc512bd14 P 98dbb90780044a349f025e385e51e5c7: No bootstrap required, opened a new log
I20260924 21:10:20.688644 14589 ts_tablet_manager.cc:1397] T 21d54bdd881a4d4888e1a1acc512bd14 P 98dbb90780044a349f025e385e51e5c7: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 21:10:20.689949 14590 tablet_bootstrap.cc:492] T 21d54bdd881a4d4888e1a1acc512bd14 P 985271ad73854c8eb773b3da78a409dc: Bootstrap starting.
I20260924 21:10:20.690495 14590 tablet_bootstrap.cc:654] T 21d54bdd881a4d4888e1a1acc512bd14 P 985271ad73854c8eb773b3da78a409dc: Neither blocks nor log segments found. Creating new log.
I20260924 21:10:20.690685 14589 raft_consensus.cc:348] T 21d54bdd881a4d4888e1a1acc512bd14 P 98dbb90780044a349f025e385e51e5c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "985271ad73854c8eb773b3da78a409dc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46821 } } peers { permanent_uuid: "98dbb90780044a349f025e385e51e5c7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44771 } } peers { permanent_uuid: "6125c714aa5640fba1a78fb9c555e3d2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41909 } }
I20260924 21:10:20.690858 14589 raft_consensus.cc:374] T 21d54bdd881a4d4888e1a1acc512bd14 P 98dbb90780044a349f025e385e51e5c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 21:10:20.690886 14589 raft_consensus.cc:729] T 21d54bdd881a4d4888e1a1acc512bd14 P 98dbb90780044a349f025e385e51e5c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98dbb90780044a349f025e385e51e5c7, State: Initialized, Role: FOLLOWER
I20260924 21:10:20.691040 14589 consensus_queue.cc:260] T 21d54bdd881a4d4888e1a1acc512bd14 P 98dbb90780044a349f025e385e51e5c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "985271ad73854c8eb773b3da78a409dc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46821 } } peers { permanent_uuid: "98dbb90780044a349f025e385e51e5c7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44771 } } peers { permanent_uuid: "6125c714aa5640fba1a78fb9c555e3d2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41909 } }
I20260924 21:10:20.691131 14590 log.cc:824] T 21d54bdd881a4d4888e1a1acc512bd14 P 985271ad73854c8eb773b3da78a409dc: Log is configured to *not* fsync() on all Append() calls
I20260924 21:10:20.691323 14589 ts_tablet_manager.cc:1428] T 21d54bdd881a4d4888e1a1acc512bd14 P 98dbb90780044a349f025e385e51e5c7: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 21:10:20.691392 14307 heartbeater.cc:503] Master 127.0.98.62:45663 was elected leader, sending a full tablet report...
I20260924 21:10:20.691512 14592 tablet_bootstrap.cc:492] T 21d54bdd881a4d4888e1a1acc512bd14 P 6125c714aa5640fba1a78fb9c555e3d2: Bootstrap starting.
I20260924 21:10:20.691962 14590 tablet_bootstrap.cc:492] T 21d54bdd881a4d4888e1a1acc512bd14 P 985271ad73854c8eb773b3da78a409dc: No bootstrap required, opened a new log
I20260924 21:10:20.692032 14590 ts_tablet_manager.cc:1397] T 21d54bdd881a4d4888e1a1acc512bd14 P 985271ad73854c8eb773b3da78a409dc: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 21:10:20.692137 14592 tablet_bootstrap.cc:654] T 21d54bdd881a4d4888e1a1acc512bd14 P 6125c714aa5640fba1a78fb9c555e3d2: Neither blocks nor log segments found. Creating new log.
I20260924 21:10:20.692668 14592 log.cc:824] T 21d54bdd881a4d4888e1a1acc512bd14 P 6125c714aa5640fba1a78fb9c555e3d2: Log is configured to *not* fsync() on all Append() calls
I20260924 21:10:20.693413 14592 tablet_bootstrap.cc:492] T 21d54bdd881a4d4888e1a1acc512bd14 P 6125c714aa5640fba1a78fb9c555e3d2: No bootstrap required, opened a new log
I20260924 21:10:20.693482 14592 ts_tablet_manager.cc:1397] T 21d54bdd881a4d4888e1a1acc512bd14 P 6125c714aa5640fba1a78fb9c555e3d2: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 21:10:20.694142 14590 raft_consensus.cc:348] T 21d54bdd881a4d4888e1a1acc512bd14 P 985271ad73854c8eb773b3da78a409dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "985271ad73854c8eb773b3da78a409dc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46821 } } peers { permanent_uuid: "98dbb90780044a349f025e385e51e5c7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44771 } } peers { permanent_uuid: "6125c714aa5640fba1a78fb9c555e3d2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41909 } }
I20260924 21:10:20.694311 14590 raft_consensus.cc:374] T 21d54bdd881a4d4888e1a1acc512bd14 P 985271ad73854c8eb773b3da78a409dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 21:10:20.694347 14590 raft_consensus.cc:729] T 21d54bdd881a4d4888e1a1acc512bd14 P 985271ad73854c8eb773b3da78a409dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 985271ad73854c8eb773b3da78a409dc, State: Initialized, Role: FOLLOWER
I20260924 21:10:20.694458 14590 consensus_queue.cc:260] T 21d54bdd881a4d4888e1a1acc512bd14 P 985271ad73854c8eb773b3da78a409dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "985271ad73854c8eb773b3da78a409dc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46821 } } peers { permanent_uuid: "98dbb90780044a349f025e385e51e5c7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44771 } } peers { permanent_uuid: "6125c714aa5640fba1a78fb9c555e3d2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41909 } }
I20260924 21:10:20.694725 14590 ts_tablet_manager.cc:1428] T 21d54bdd881a4d4888e1a1acc512bd14 P 985271ad73854c8eb773b3da78a409dc: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 21:10:20.694730 14575 heartbeater.cc:503] Master 127.0.98.62:45663 was elected leader, sending a full tablet report...
I20260924 21:10:20.695480 14592 raft_consensus.cc:348] T 21d54bdd881a4d4888e1a1acc512bd14 P 6125c714aa5640fba1a78fb9c555e3d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "985271ad73854c8eb773b3da78a409dc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46821 } } peers { permanent_uuid: "98dbb90780044a349f025e385e51e5c7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44771 } } peers { permanent_uuid: "6125c714aa5640fba1a78fb9c555e3d2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41909 } }
I20260924 21:10:20.695662 14592 raft_consensus.cc:374] T 21d54bdd881a4d4888e1a1acc512bd14 P 6125c714aa5640fba1a78fb9c555e3d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 21:10:20.695713 14592 raft_consensus.cc:729] T 21d54bdd881a4d4888e1a1acc512bd14 P 6125c714aa5640fba1a78fb9c555e3d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6125c714aa5640fba1a78fb9c555e3d2, State: Initialized, Role: FOLLOWER
I20260924 21:10:20.695825 14592 consensus_queue.cc:260] T 21d54bdd881a4d4888e1a1acc512bd14 P 6125c714aa5640fba1a78fb9c555e3d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "985271ad73854c8eb773b3da78a409dc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46821 } } peers { permanent_uuid: "98dbb90780044a349f025e385e51e5c7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44771 } } peers { permanent_uuid: "6125c714aa5640fba1a78fb9c555e3d2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41909 } }
I20260924 21:10:20.696076 14592 ts_tablet_manager.cc:1428] T 21d54bdd881a4d4888e1a1acc512bd14 P 6125c714aa5640fba1a78fb9c555e3d2: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 21:10:20.696141 14441 heartbeater.cc:503] Master 127.0.98.62:45663 was elected leader, sending a full tablet report...
W20260924 21:10:20.758621 14442 tablet.cc:2369] T 21d54bdd881a4d4888e1a1acc512bd14 P 6125c714aa5640fba1a78fb9c555e3d2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 21:10:20.799398 14597 raft_consensus.cc:482] T 21d54bdd881a4d4888e1a1acc512bd14 P 6125c714aa5640fba1a78fb9c555e3d2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 21:10:20.799563 14597 raft_consensus.cc:504] T 21d54bdd881a4d4888e1a1acc512bd14 P 6125c714aa5640fba1a78fb9c555e3d2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "985271ad73854c8eb773b3da78a409dc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46821 } } peers { permanent_uuid: "98dbb90780044a349f025e385e51e5c7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44771 } } peers { permanent_uuid: "6125c714aa5640fba1a78fb9c555e3d2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41909 } }
I20260924 21:10:20.799925 14597 leader_election.cc:288] T 21d54bdd881a4d4888e1a1acc512bd14 P 6125c714aa5640fba1a78fb9c555e3d2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 985271ad73854c8eb773b3da78a409dc (127.0.98.3:46821), 98dbb90780044a349f025e385e51e5c7 (127.0.98.1:44771)
I20260924 21:10:20.802614 14443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:10:20.800073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44771 (local address 127.0.98.2:59877)
0924 21:10:20.800120 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0924 21:10:20.800130 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 21:10:20.800166 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 21:10:20.800388 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 21:10:20.800389 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 21:10:20.800390 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 21:10:20.800469 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:10:20.800471 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:10:20.801097 (+ 626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:10:20.801098 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:10:20.801777 (+ 679us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:10:20.801779 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:10:20.802366 (+ 587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:10:20.802367 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:10:20.802446 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:10:20.802449 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 21:10:20.802515 (+ 66us) client_negotiation.cc:240] Negotiation successful
0924 21:10:20.802554 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260924 21:10:20.802614 14586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:10:20.800129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:59877 (local address 127.0.98.1:44771)
0924 21:10:20.800164 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0924 21:10:20.800166 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 21:10:20.800174 (+ 8us) server_negotiation.cc:407] Connection header received
0924 21:10:20.800226 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:10:20.800227 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:10:20.800260 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:10:20.800366 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 21:10:20.800546 (+ 180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:10:20.801014 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:10:20.801909 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:10:20.802273 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:10:20.802368 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:10:20.802385 (+ 17us) server_negotiation.cc:1035] Waiting for connection context
0924 21:10:20.802521 (+ 136us) server_negotiation.cc:299] Negotiation successful
0924 21:10:20.802557 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260924 21:10:20.802738 14587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:10:20.800222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:42371 (local address 127.0.98.3:46821)
0924 21:10:20.800287 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0924 21:10:20.800289 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 21:10:20.800410 (+ 121us) server_negotiation.cc:407] Connection header received
0924 21:10:20.800481 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:10:20.800482 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:10:20.800501 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:10:20.800560 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 21:10:20.800711 (+ 151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:10:20.801165 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:10:20.802062 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:10:20.802432 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:10:20.802510 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:10:20.802518 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0924 21:10:20.802649 (+ 131us) server_negotiation.cc:299] Negotiation successful
0924 21:10:20.802683 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260924 21:10:20.802752 14598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:10:20.800150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:46821 (local address 127.0.98.2:42371)
0924 21:10:20.800321 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0924 21:10:20.800341 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 21:10:20.800417 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 21:10:20.800565 (+ 148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 21:10:20.800566 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 21:10:20.800567 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 21:10:20.800641 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:10:20.800642 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:10:20.801248 (+ 606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:10:20.801249 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:10:20.801989 (+ 740us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:10:20.801991 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:10:20.802511 (+ 520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:10:20.802512 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:10:20.802581 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:10:20.802584 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 21:10:20.802644 (+ 60us) client_negotiation.cc:240] Negotiation successful
0924 21:10:20.802678 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"mutex_wait_us":1,"thread_start_us":99,"threads_started":1}
I20260924 21:10:20.803109 14529 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21d54bdd881a4d4888e1a1acc512bd14" candidate_uuid: "6125c714aa5640fba1a78fb9c555e3d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "985271ad73854c8eb773b3da78a409dc" is_pre_election: true
I20260924 21:10:20.803122 14261 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21d54bdd881a4d4888e1a1acc512bd14" candidate_uuid: "6125c714aa5640fba1a78fb9c555e3d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98dbb90780044a349f025e385e51e5c7" is_pre_election: true
I20260924 21:10:20.803267 14261 raft_consensus.cc:2454] T 21d54bdd881a4d4888e1a1acc512bd14 P 98dbb90780044a349f025e385e51e5c7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6125c714aa5640fba1a78fb9c555e3d2 in term 0.
I20260924 21:10:20.803251 14529 raft_consensus.cc:2454] T 21d54bdd881a4d4888e1a1acc512bd14 P 985271ad73854c8eb773b3da78a409dc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6125c714aa5640fba1a78fb9c555e3d2 in term 0.
I20260924 21:10:20.803514 14329 leader_election.cc:302] T 21d54bdd881a4d4888e1a1acc512bd14 P 6125c714aa5640fba1a78fb9c555e3d2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 6125c714aa5640fba1a78fb9c555e3d2, 985271ad73854c8eb773b3da78a409dc, 98dbb90780044a349f025e385e51e5c7; no voters:
I20260924 21:10:20.803643 14597 raft_consensus.cc:2781] T 21d54bdd881a4d4888e1a1acc512bd14 P 6125c714aa5640fba1a78fb9c555e3d2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 21:10:20.803689 14597 raft_consensus.cc:482] T 21d54bdd881a4d4888e1a1acc512bd14 P 6125c714aa5640fba1a78fb9c555e3d2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 21:10:20.803714 14597 raft_consensus.cc:3037] T 21d54bdd881a4d4888e1a1acc512bd14 P 6125c714aa5640fba1a78fb9c555e3d2 [term 0 FOLLOWER]: Advancing to term 1
I20260924 21:10:20.804296 14597 raft_consensus.cc:504] T 21d54bdd881a4d4888e1a1acc512bd14 P 6125c714aa5640fba1a78fb9c555e3d2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "985271ad73854c8eb773b3da78a409dc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46821 } } peers { permanent_uuid: "98dbb90780044a349f025e385e51e5c7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44771 } } peers { permanent_uuid: "6125c714aa5640fba1a78fb9c555e3d2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41909 } }
I20260924 21:10:20.804487 14597 leader_election.cc:288] T 21d54bdd881a4d4888e1a1acc512bd14 P 6125c714aa5640fba1a78fb9c555e3d2 [CANDIDATE]: Term 1 election: Requested vote from peers 985271ad73854c8eb773b3da78a409dc (127.0.98.3:46821), 98dbb90780044a349f025e385e51e5c7 (127.0.98.1:44771)
I20260924 21:10:20.804565 14261 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21d54bdd881a4d4888e1a1acc512bd14" candidate_uuid: "6125c714aa5640fba1a78fb9c555e3d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98dbb90780044a349f025e385e51e5c7"
I20260924 21:10:20.804633 14261 raft_consensus.cc:3037] T 21d54bdd881a4d4888e1a1acc512bd14 P 98dbb90780044a349f025e385e51e5c7 [term 0 FOLLOWER]: Advancing to term 1
I20260924 21:10:20.805279 14261 raft_consensus.cc:2454] T 21d54bdd881a4d4888e1a1acc512bd14 P 98dbb90780044a349f025e385e51e5c7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6125c714aa5640fba1a78fb9c555e3d2 in term 1.
I20260924 21:10:20.805400 14529 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21d54bdd881a4d4888e1a1acc512bd14" candidate_uuid: "6125c714aa5640fba1a78fb9c555e3d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "985271ad73854c8eb773b3da78a409dc"
I20260924 21:10:20.805452 14331 leader_election.cc:302] T 21d54bdd881a4d4888e1a1acc512bd14 P 6125c714aa5640fba1a78fb9c555e3d2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6125c714aa5640fba1a78fb9c555e3d2, 98dbb90780044a349f025e385e51e5c7; no voters:
I20260924 21:10:20.805487 14529 raft_consensus.cc:3037] T 21d54bdd881a4d4888e1a1acc512bd14 P 985271ad73854c8eb773b3da78a409dc [term 0 FOLLOWER]: Advancing to term 1
I20260924 21:10:20.805536 14597 raft_consensus.cc:2781] T 21d54bdd881a4d4888e1a1acc512bd14 P 6125c714aa5640fba1a78fb9c555e3d2 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 21:10:20.805660 14597 raft_consensus.cc:686] T 21d54bdd881a4d4888e1a1acc512bd14 P 6125c714aa5640fba1a78fb9c555e3d2 [term 1 LEADER]: Becoming Leader. State: Replica: 6125c714aa5640fba1a78fb9c555e3d2, State: Running, Role: LEADER
I20260924 21:10:20.805886 14597 consensus_queue.cc:237] T 21d54bdd881a4d4888e1a1acc512bd14 P 6125c714aa5640fba1a78fb9c555e3d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "985271ad73854c8eb773b3da78a409dc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46821 } } peers { permanent_uuid: "98dbb90780044a349f025e385e51e5c7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44771 } } peers { permanent_uuid: "6125c714aa5640fba1a78fb9c555e3d2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41909 } }
I20260924 21:10:20.806344 14529 raft_consensus.cc:2454] T 21d54bdd881a4d4888e1a1acc512bd14 P 985271ad73854c8eb773b3da78a409dc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6125c714aa5640fba1a78fb9c555e3d2 in term 1.
I20260924 21:10:20.806656 14116 catalog_manager.cc:5495] T 21d54bdd881a4d4888e1a1acc512bd14 P 6125c714aa5640fba1a78fb9c555e3d2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6125c714aa5640fba1a78fb9c555e3d2 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "6125c714aa5640fba1a78fb9c555e3d2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "985271ad73854c8eb773b3da78a409dc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46821 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "98dbb90780044a349f025e385e51e5c7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44771 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6125c714aa5640fba1a78fb9c555e3d2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41909 } health_report { overall_health: HEALTHY } } }
I20260924 21:10:20.824986 14588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:10:20.822968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55528 (local address 127.0.98.2:41909)
0924 21:10:20.823026 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0924 21:10:20.823028 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 21:10:20.823035 (+ 7us) server_negotiation.cc:407] Connection header received
0924 21:10:20.823097 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:10:20.823098 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:10:20.823116 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:10:20.823187 (+ 71us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 21:10:20.823296 (+ 109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:10:20.823680 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:10:20.824322 (+ 642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:10:20.824498 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:10:20.824563 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:10:20.824685 (+ 122us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 21:10:20.824774 (+ 89us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 21:10:20.824853 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0924 21:10:20.824886 (+ 33us) server_negotiation.cc:299] Negotiation successful
0924 21:10:20.824902 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260924 21:10:20.826519 14261 raft_consensus.cc:1261] T 21d54bdd881a4d4888e1a1acc512bd14 P 98dbb90780044a349f025e385e51e5c7 [term 1 FOLLOWER]: Refusing update from remote peer 6125c714aa5640fba1a78fb9c555e3d2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 21:10:20.826761 14529 raft_consensus.cc:1261] T 21d54bdd881a4d4888e1a1acc512bd14 P 985271ad73854c8eb773b3da78a409dc [term 1 FOLLOWER]: Refusing update from remote peer 6125c714aa5640fba1a78fb9c555e3d2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 21:10:20.826821 14599 consensus_queue.cc:1035] T 21d54bdd881a4d4888e1a1acc512bd14 P 6125c714aa5640fba1a78fb9c555e3d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "98dbb90780044a349f025e385e51e5c7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44771 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 21:10:20.826965 14599 consensus_queue.cc:1035] T 21d54bdd881a4d4888e1a1acc512bd14 P 6125c714aa5640fba1a78fb9c555e3d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "985271ad73854c8eb773b3da78a409dc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 21:10:20.828653 14602 mvcc.cc:205] Tried to move back new op lower bound from 7333004168503205888 to 7333004168421552128. Current Snapshot: MvccSnapshot[applied={T|T < 7333004168503205888}]
I20260924 21:10:20.828913 14603 mvcc.cc:205] Tried to move back new op lower bound from 7333004168503205888 to 7333004168421552128. Current Snapshot: MvccSnapshot[applied={T|T < 7333004168503205888}]
I20260924 21:10:20.829352 14604 mvcc.cc:205] Tried to move back new op lower bound from 7333004168503205888 to 7333004168421552128. Current Snapshot: MvccSnapshot[applied={T|T < 7333004168503205888}]
I20260924 21:10:20.831138 14116 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47424:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 21:10:20.831230 14116 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47424:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 21:10:20.831943 14116 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4e3cd7d6c2b4441093bfb5a89a0767ae: Sending DeleteTablet for 3 replicas of tablet 21d54bdd881a4d4888e1a1acc512bd14
I20260924 21:10:20.832216 14509 tablet_service.cc:1514] Processing DeleteTablet for tablet 21d54bdd881a4d4888e1a1acc512bd14 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 21:10:20 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:37544
I20260924 21:10:20.832279 14241 tablet_service.cc:1514] Processing DeleteTablet for tablet 21d54bdd881a4d4888e1a1acc512bd14 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 21:10:20 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:57138
I20260924 21:10:20.832455 14613 tablet_replica.cc:330] T 21d54bdd881a4d4888e1a1acc512bd14 P 98dbb90780044a349f025e385e51e5c7: stopping tablet replica
I20260924 21:10:20.832455 14375 tablet_service.cc:1514] Processing DeleteTablet for tablet 21d54bdd881a4d4888e1a1acc512bd14 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 21:10:20 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:55516
I20260924 21:10:20.832535 14613 raft_consensus.cc:2229] T 21d54bdd881a4d4888e1a1acc512bd14 P 98dbb90780044a349f025e385e51e5c7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 21:10:20.832620 14613 raft_consensus.cc:2258] T 21d54bdd881a4d4888e1a1acc512bd14 P 98dbb90780044a349f025e385e51e5c7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 21:10:20.832647 14614 tablet_replica.cc:330] T 21d54bdd881a4d4888e1a1acc512bd14 P 6125c714aa5640fba1a78fb9c555e3d2: stopping tablet replica
I20260924 21:10:20.832849 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAzNNjS/build/release/bin/kudu with pid 14178
I20260924 21:10:20.832881 14614 raft_consensus.cc:2229] T 21d54bdd881a4d4888e1a1acc512bd14 P 6125c714aa5640fba1a78fb9c555e3d2 [term 1 LEADER]: Raft consensus shutting down.
I20260924 21:10:20.833007 14614 raft_consensus.cc:2258] T 21d54bdd881a4d4888e1a1acc512bd14 P 6125c714aa5640fba1a78fb9c555e3d2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 21:10:20.833017 14613 ts_tablet_manager.cc:1905] T 21d54bdd881a4d4888e1a1acc512bd14 P 98dbb90780044a349f025e385e51e5c7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 21:10:20.833415 14614 ts_tablet_manager.cc:1905] T 21d54bdd881a4d4888e1a1acc512bd14 P 6125c714aa5640fba1a78fb9c555e3d2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 21:10:20.834633 14612 tablet_replica.cc:330] T 21d54bdd881a4d4888e1a1acc512bd14 P 985271ad73854c8eb773b3da78a409dc: stopping tablet replica
I20260924 21:10:20.834725 14612 raft_consensus.cc:2229] T 21d54bdd881a4d4888e1a1acc512bd14 P 985271ad73854c8eb773b3da78a409dc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 21:10:20.834817 14612 raft_consensus.cc:2258] T 21d54bdd881a4d4888e1a1acc512bd14 P 985271ad73854c8eb773b3da78a409dc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 21:10:20.835145 14612 ts_tablet_manager.cc:1905] T 21d54bdd881a4d4888e1a1acc512bd14 P 985271ad73854c8eb773b3da78a409dc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 21:10:20.836802 14612 ts_tablet_manager.cc:1918] T 21d54bdd881a4d4888e1a1acc512bd14 P 985271ad73854c8eb773b3da78a409dc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 21:10:20.836874 14612 log.cc:1196] T 21d54bdd881a4d4888e1a1acc512bd14 P 985271ad73854c8eb773b3da78a409dc: Deleting WAL directory at /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790284076800039-392-0/minicluster-data/ts-2/wal/wals/21d54bdd881a4d4888e1a1acc512bd14
I20260924 21:10:20.837143 14612 ts_tablet_manager.cc:1939] T 21d54bdd881a4d4888e1a1acc512bd14 P 985271ad73854c8eb773b3da78a409dc: Deleting consensus metadata
I20260924 21:10:20.837688 14101 catalog_manager.cc:4841] TS 985271ad73854c8eb773b3da78a409dc (127.0.98.3:46821): tablet 21d54bdd881a4d4888e1a1acc512bd14 (table test-table [id=7dc021814272415dbe159263a5ae9e75]) successfully deleted
W20260924 21:10:20.839061 14103 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:44771 (error 108)
I20260924 21:10:20.839246 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAzNNjS/build/release/bin/kudu with pid 14312
I20260924 21:10:20.839717 14583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:10:20.839366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44771 (local address 127.0.0.1:57150)
0924 21:10:20.839654 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0924 21:10:20.839685 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:44771: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":273}
W20260924 21:10:20.839900 14103 catalog_manager.cc:4584] TS 98dbb90780044a349f025e385e51e5c7 (127.0.98.1:44771): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 21d54bdd881a4d4888e1a1acc512bd14: Network error: Client connection negotiation failed: client connection to 127.0.98.1:44771: connect: Connection refused (error 111)
I20260924 21:10:20.843439 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAzNNjS/build/release/bin/kudu with pid 14446
I20260924 21:10:20.843472 14583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:10:20.843381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41909 (local address 127.0.0.1:55538)
0924 21:10:20.843428 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0924 21:10:20.843453 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:41909: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
I20260924 21:10:20.847131 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAzNNjS/build/release/bin/kudu with pid 14084
I20260924 21:10:20.851997 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAzNNjS/build/release/bin/kudu
/tmp/dist-test-taskAzNNjS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790284076800039-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790284076800039-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790284076800039-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790284076800039-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45663
--webserver_interface=127.0.98.62
--webserver_port=41717
--keytab_file=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790284076800039-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35231
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45663
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790284076800039-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790284076800039-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790284076800039-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 21:10:20.921854 14616 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 21:10:20.922013 14616 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 21:10:20.922032 14616 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 21:10:20.922991 14616 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 21:10:20.923032 14616 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 21:10:20.923044 14616 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 21:10:20.923055 14616 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 21:10:20.923065 14616 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 21:10:20.924367 14616 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35231
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790284076800039-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790284076800039-392-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45663
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790284076800039-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45663
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790284076800039-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=41717
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14616
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790284076800039-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 19:49:06 UTC on ccycloud.quasar-vhqjoc.root.comops.site
build id 4670864
I20260924 21:10:20.924580 14616 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 21:10:20.924767 14616 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 21:10:20.931908 14623 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 21:10:20.931945 14625 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 21:10:20.931988 14622 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 21:10:20.932082 14616 server_base.cc:1034] running on GCE node
I20260924 21:10:20.932328 14616 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 21:10:20.932616 14616 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 21:10:20.933763 14616 hybrid_clock.cc:648] HybridClock initialized: now 1790284220933748 us; error 30 us; skew 500 ppm
Sep 24 21:10:20 dist-test-slave-c7tk krb5kdc[14071](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284220, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 21:10:20.935810 14616 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 21:10:20.936213 14616 webserver.cc:460] Webserver started at http://127.0.98.62:41717/ using document root <none> and password file <none>
I20260924 21:10:20.936415 14616 fs_manager.cc:359] Metadata directory not provided
I20260924 21:10:20.936461 14616 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 21:10:20.937549 14616 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 21:10:20.938136 14632 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 21:10:20.938292 14616 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 21:10:20.938354 14616 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790284076800039-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790284076800039-392-0/minicluster-data/master-0/wal
uuid: "4e3cd7d6c2b4441093bfb5a89a0767ae"
format_stamp: "Formatted at 2026-09-24 21:10:20 on dist-test-slave-c7tk"
server_key: "e7de828d96504a2c38339d0e70f025a0"
server_key_iv: "dcbce15300373d651b4546345bfa50e7"
server_key_version: "encryptionkey@0"
I20260924 21:10:20.938666 14616 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790284076800039-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790284076800039-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790284076800039-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 21:10:20.961436 14616 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 21:10:20.963037 14616 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 21:10:20.963164 14616 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 21:10:20.966826 14616 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45663
I20260924 21:10:20.966898 14684 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45663 every 8 connection(s)
I20260924 21:10:20.967209 14616 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790284076800039-392-0/minicluster-data/master-0/data/info.pb
I20260924 21:10:20.968605 14685 sys_catalog.cc:263] Verifying existing consensus state
I20260924 21:10:20.969158 14685 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4e3cd7d6c2b4441093bfb5a89a0767ae: Bootstrap starting.
I20260924 21:10:20.976135 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAzNNjS/build/release/bin/kudu as pid 14616
I20260924 21:10:20.976401 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAzNNjS/build/release/bin/kudu
/tmp/dist-test-taskAzNNjS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790284076800039-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790284076800039-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790284076800039-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790284076800039-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:44771
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:45663
--webserver_port=38341
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790284076800039-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35231
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790284076800039-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790284076800039-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790284076800039-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 21:10:20.976754 14685 log.cc:824] T 00000000000000000000000000000000 P 4e3cd7d6c2b4441093bfb5a89a0767ae: Log is configured to *not* fsync() on all Append() calls
I20260924 21:10:20.978914 14685 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4e3cd7d6c2b4441093bfb5a89a0767ae: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 21:10:20.979140 14685 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4e3cd7d6c2b4441093bfb5a89a0767ae: Bootstrap complete.
I20260924 21:10:20.981120 14685 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4e3cd7d6c2b4441093bfb5a89a0767ae [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e3cd7d6c2b4441093bfb5a89a0767ae" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45663 } }
I20260924 21:10:20.981350 14685 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4e3cd7d6c2b4441093bfb5a89a0767ae [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e3cd7d6c2b4441093bfb5a89a0767ae, State: Initialized, Role: FOLLOWER
I20260924 21:10:20.981539 14685 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4e3cd7d6c2b4441093bfb5a89a0767ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e3cd7d6c2b4441093bfb5a89a0767ae" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45663 } }
I20260924 21:10:20.981606 14685 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4e3cd7d6c2b4441093bfb5a89a0767ae [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 21:10:20.981652 14685 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4e3cd7d6c2b4441093bfb5a89a0767ae [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 21:10:20.981691 14685 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4e3cd7d6c2b4441093bfb5a89a0767ae [term 1 FOLLOWER]: Advancing to term 2
I20260924 21:10:20.982529 14685 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4e3cd7d6c2b4441093bfb5a89a0767ae [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e3cd7d6c2b4441093bfb5a89a0767ae" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45663 } }
I20260924 21:10:20.982630 14685 leader_election.cc:302] T 00000000000000000000000000000000 P 4e3cd7d6c2b4441093bfb5a89a0767ae [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4e3cd7d6c2b4441093bfb5a89a0767ae; no voters:
I20260924 21:10:20.982760 14685 leader_election.cc:288] T 00000000000000000000000000000000 P 4e3cd7d6c2b4441093bfb5a89a0767ae [CANDIDATE]: Term 2 election: Requested vote from peers
I20260924 21:10:20.982788 14689 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4e3cd7d6c2b4441093bfb5a89a0767ae [term 2 FOLLOWER]: Leader election won for term 2
I20260924 21:10:20.982893 14689 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4e3cd7d6c2b4441093bfb5a89a0767ae [term 2 LEADER]: Becoming Leader. State: Replica: 4e3cd7d6c2b4441093bfb5a89a0767ae, State: Running, Role: LEADER
I20260924 21:10:20.982963 14689 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4e3cd7d6c2b4441093bfb5a89a0767ae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e3cd7d6c2b4441093bfb5a89a0767ae" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45663 } }
I20260924 21:10:20.982986 14685 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4e3cd7d6c2b4441093bfb5a89a0767ae [sys.catalog]: configured and running, proceeding with master startup.
I20260924 21:10:20.984248 14691 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4e3cd7d6c2b4441093bfb5a89a0767ae [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4e3cd7d6c2b4441093bfb5a89a0767ae. Latest consensus state: current_term: 2 leader_uuid: "4e3cd7d6c2b4441093bfb5a89a0767ae" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e3cd7d6c2b4441093bfb5a89a0767ae" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45663 } } }
I20260924 21:10:20.984306 14691 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4e3cd7d6c2b4441093bfb5a89a0767ae [sys.catalog]: This master's current role is: LEADER
I20260924 21:10:20.984125 14690 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4e3cd7d6c2b4441093bfb5a89a0767ae [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "4e3cd7d6c2b4441093bfb5a89a0767ae" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e3cd7d6c2b4441093bfb5a89a0767ae" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45663 } } }
I20260924 21:10:20.984493 14690 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4e3cd7d6c2b4441093bfb5a89a0767ae [sys.catalog]: This master's current role is: LEADER
I20260924 21:10:20.984570 14703 catalog_manager.cc:1259] Loaded cluster ID: 29f9a19ac9f14577a5de672c45a0000e
I20260924 21:10:20.984613 14703 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 4e3cd7d6c2b4441093bfb5a89a0767ae: loading cluster ID for follower catalog manager: success
W20260924 21:10:20.984822 14703 catalog_manager.cc:1576] T 00000000000000000000000000000000 P 4e3cd7d6c2b4441093bfb5a89a0767ae: acquiring CA information for follower catalog manager: Runtime error: could not decrypt private key with the password returned by the configured command: unable to load data from memory: error reading private key: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D07803A:asn1 encoding routines:ASN1_ITEM_EX_D2I:nested asn1 error:tasn_dec.c:386:Type=X509_ALGOR error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=algor, Type=X509_SIG
W20260924 21:10:20.984866 14703 catalog_manager.cc:873] Runtime error: could not decrypt private key with the password returned by the configured command: unable to load data from memory: error reading private key: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D07803A:asn1 encoding routines:ASN1_ITEM_EX_D2I:nested asn1 error:tasn_dec.c:386:Type=X509_ALGOR error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=algor, Type=X509_SIG: failed to prepare follower catalog manager, will retry
I20260924 21:10:20.985001 14706 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 21:10:20.985452 14706 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 21:10:20.985553 14706 catalog_manager.cc:1259] Loaded cluster ID: 29f9a19ac9f14577a5de672c45a0000e
I20260924 21:10:20.985592 14706 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 21:10:20.985750 14706 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260924 21:10:20.990200 14706 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260924 21:10:20.992338 14706 catalog_manager.cc:1504] Loading token signing keys...
I20260924 21:10:20.992501 14706 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 4e3cd7d6c2b4441093bfb5a89a0767ae: Loaded TSK: 0
I20260924 21:10:20.992678 14706 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260924 21:10:21.050724 14688 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 21:10:21.050904 14688 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 21:10:21.050926 14688 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 21:10:21.051996 14688 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 21:10:21.052030 14688 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 21:10:21.052053 14688 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 21:10:21.053328 14688 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35231
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790284076800039-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790284076800039-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790284076800039-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:44771
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790284076800039-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=38341
--tserver_master_addrs=127.0.98.62:45663
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14688
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790284076800039-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 19:49:06 UTC on ccycloud.quasar-vhqjoc.root.comops.site
build id 4670864
I20260924 21:10:21.053536 14688 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 21:10:21.053745 14688 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 21:10:21.058890 14713 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 21:10:21.059028 14712 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 21:10:21.058893 14715 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 21:10:21.059033 14688 server_base.cc:1034] running on GCE node
I20260924 21:10:21.059370 14688 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 21:10:21.059623 14688 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 21:10:21.060789 14688 hybrid_clock.cc:648] HybridClock initialized: now 1790284221060771 us; error 25 us; skew 500 ppm
Sep 24 21:10:21 dist-test-slave-c7tk krb5kdc[14071](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284221, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 21:10:21.063457 14688 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 21:10:21.063858 14688 webserver.cc:460] Webserver started at http://127.0.98.1:38341/ using document root <none> and password file <none>
I20260924 21:10:21.064057 14688 fs_manager.cc:359] Metadata directory not provided
I20260924 21:10:21.064128 14688 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 21:10:21.065282 14688 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 21:10:21.066027 14722 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 21:10:21.066185 14688 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 21:10:21.066279 14688 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790284076800039-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790284076800039-392-0/minicluster-data/ts-0/wal
uuid: "98dbb90780044a349f025e385e51e5c7"
format_stamp: "Formatted at 2026-09-24 21:10:20 on dist-test-slave-c7tk"
server_key: "ad8242e15fbb6b838111c50417b89741"
server_key_iv: "a745b8c1045ad53db9fd3d021a833161"
server_key_version: "encryptionkey@0"
I20260924 21:10:21.066591 14688 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790284076800039-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790284076800039-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790284076800039-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 21:10:21.084771 14688 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 21:10:21.086678 14688 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 21:10:21.086817 14688 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 21:10:21.087018 14688 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 21:10:21.087462 14729 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260924 21:10:21.088778 14688 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260924 21:10:21.088821 14688 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260924 21:10:21.088868 14688 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260924 21:10:21.089396 14688 ts_tablet_manager.cc:610] Registered 1 tablets
I20260924 21:10:21.089434 14688 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260924 21:10:21.089468 14729 tablet_bootstrap.cc:492] T 21d54bdd881a4d4888e1a1acc512bd14 P 98dbb90780044a349f025e385e51e5c7: Bootstrap starting.
I20260924 21:10:21.091255 14729 log.cc:824] T 21d54bdd881a4d4888e1a1acc512bd14 P 98dbb90780044a349f025e385e51e5c7: Log is configured to *not* fsync() on all Append() calls
I20260924 21:10:21.094147 14729 tablet_bootstrap.cc:492] T 21d54bdd881a4d4888e1a1acc512bd14 P 98dbb90780044a349f025e385e51e5c7: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 21:10:21.094421 14729 tablet_bootstrap.cc:492] T 21d54bdd881a4d4888e1a1acc512bd14 P 98dbb90780044a349f025e385e51e5c7: Bootstrap complete.
I20260924 21:10:21.094565 14729 ts_tablet_manager.cc:1397] T 21d54bdd881a4d4888e1a1acc512bd14 P 98dbb90780044a349f025e385e51e5c7: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.003s
I20260924 21:10:21.095908 14688 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:44771
I20260924 21:10:21.095940 14837 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:44771 every 8 connection(s)
I20260924 21:10:21.096212 14688 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790284076800039-392-0/minicluster-data/ts-0/data/info.pb
I20260924 21:10:21.096413 14729 raft_consensus.cc:348] T 21d54bdd881a4d4888e1a1acc512bd14 P 98dbb90780044a349f025e385e51e5c7 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "985271ad73854c8eb773b3da78a409dc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46821 } } peers { permanent_uuid: "98dbb90780044a349f025e385e51e5c7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44771 } } peers { permanent_uuid: "6125c714aa5640fba1a78fb9c555e3d2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41909 } }
I20260924 21:10:21.096709 14729 raft_consensus.cc:729] T 21d54bdd881a4d4888e1a1acc512bd14 P 98dbb90780044a349f025e385e51e5c7 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98dbb90780044a349f025e385e51e5c7, State: Initialized, Role: FOLLOWER
I20260924 21:10:21.096848 14729 consensus_queue.cc:260] T 21d54bdd881a4d4888e1a1acc512bd14 P 98dbb90780044a349f025e385e51e5c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "985271ad73854c8eb773b3da78a409dc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46821 } } peers { permanent_uuid: "98dbb90780044a349f025e385e51e5c7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44771 } } peers { permanent_uuid: "6125c714aa5640fba1a78fb9c555e3d2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41909 } }
I20260924 21:10:21.097167 14729 ts_tablet_manager.cc:1428] T 21d54bdd881a4d4888e1a1acc512bd14 P 98dbb90780044a349f025e385e51e5c7: Time spent starting tablet: real 0.003s user 0.002s sys 0.002s
I20260924 21:10:21.101146 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAzNNjS/build/release/bin/kudu as pid 14688
I20260924 21:10:21.101425 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAzNNjS/build/release/bin/kudu
/tmp/dist-test-taskAzNNjS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790284076800039-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790284076800039-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790284076800039-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790284076800039-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:41909
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:45663
--webserver_port=46117
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790284076800039-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35231
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790284076800039-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790284076800039-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790284076800039-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 21:10:21 dist-test-slave-c7tk krb5kdc[14071](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284221, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 21:10:21.104887 14840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:10:21.096852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45663 (local address 127.0.98.1:40865)
0924 21:10:21.096989 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0924 21:10:21.097021 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0924 21:10:21.097136 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 21:10:21.097593 (+ 457us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 21:10:21.097595 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 21:10:21.097815 (+ 220us) client_negotiation.cc:189] Negotiated authn=SASL
0924 21:10:21.097971 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:10:21.097974 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:10:21.098823 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:10:21.098823 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:10:21.099452 (+ 629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:10:21.099454 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:10:21.099877 (+ 423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:10:21.099877 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:10:21.099953 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:10:21.100283 (+ 330us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 21:10:21.100289 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0924 21:10:21.102054 (+ 1765us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 21:10:21.103648 (+ 1594us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 21:10:21.103654 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 21:10:21.103656 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 21:10:21.103841 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 21:10:21.104087 (+ 246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 21:10:21.104088 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 21:10:21.104088 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 21:10:21.104166 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 21:10:21.104526 (+ 360us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 21:10:21.104530 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 21:10:21.104613 (+ 83us) client_negotiation.cc:769] Sending connection context
0924 21:10:21.104677 (+ 64us) client_negotiation.cc:240] Negotiation successful
0924 21:10:21.104769 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":80,"threads_started":1}
I20260924 21:10:21.105093 14841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:10:21.096870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:40865 (local address 127.0.98.62:45663)
0924 21:10:21.097321 (+ 451us) server_negotiation.cc:206] Beginning negotiation
0924 21:10:21.097327 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 21:10:21.097348 (+ 21us) server_negotiation.cc:407] Connection header received
0924 21:10:21.097432 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:10:21.097436 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:10:21.097469 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:10:21.097562 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:10:21.098050 (+ 488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:10:21.098741 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:10:21.099514 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:10:21.099815 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:10:21.099886 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:10:21.102136 (+ 2250us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 21:10:21.102140 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 21:10:21.102140 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 21:10:21.102149 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 21:10:21.103563 (+ 1414us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:10:21.103922 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:10:21.103924 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:10:21.103925 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:10:21.104026 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:10:21.104222 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:10:21.104223 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:10:21.104223 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:10:21.104465 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 21:10:21.104540 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0924 21:10:21.104750 (+ 210us) server_negotiation.cc:299] Negotiation successful
0924 21:10:21.104889 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":410,"thread_start_us":72,"threads_started":1}
I20260924 21:10:21.105719 14838 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45663
I20260924 21:10:21.105822 14838 heartbeater.cc:464] Registering TS with master...
I20260924 21:10:21.105959 14838 heartbeater.cc:511] Master 127.0.98.62:45663 requested a full tablet report, sending...
I20260924 21:10:21.106560 14649 ts_manager.cc:194] Registered new tserver with Master: 98dbb90780044a349f025e385e51e5c7 (127.0.98.1:44771)
I20260924 21:10:21.107620 14649 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:40865
I20260924 21:10:21.108359 14838 heartbeater.cc:503] Master 127.0.98.62:45663 was elected leader, sending a full tablet report...
Sep 24 21:10:21 dist-test-slave-c7tk krb5kdc[14071](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284220, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260924 21:10:21.113981 14844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:10:21.107080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44771 (local address 127.0.0.1:57164)
0924 21:10:21.107290 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0924 21:10:21.107310 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 21:10:21.107384 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 21:10:21.107554 (+ 170us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 21:10:21.107557 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 21:10:21.107824 (+ 267us) client_negotiation.cc:189] Negotiated authn=SASL
0924 21:10:21.107892 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:10:21.107894 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:10:21.108827 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:10:21.108828 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:10:21.109302 (+ 474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:10:21.109303 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:10:21.109719 (+ 416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:10:21.109722 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:10:21.109783 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:10:21.110174 (+ 391us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 21:10:21.110178 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 21:10:21.111556 (+ 1378us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 21:10:21.113108 (+ 1552us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 21:10:21.113113 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 21:10:21.113114 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 21:10:21.113264 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 21:10:21.113443 (+ 179us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 21:10:21.113444 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 21:10:21.113444 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 21:10:21.113486 (+ 42us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 21:10:21.113721 (+ 235us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 21:10:21.113725 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 21:10:21.113756 (+ 31us) client_negotiation.cc:769] Sending connection context
0924 21:10:21.113860 (+ 104us) client_negotiation.cc:240] Negotiation successful
0924 21:10:21.113930 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":69,"threads_started":1}
I20260924 21:10:21.114044 14845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:10:21.107123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57164 (local address 127.0.98.1:44771)
0924 21:10:21.107247 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0924 21:10:21.107249 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 21:10:21.107370 (+ 121us) server_negotiation.cc:407] Connection header received
0924 21:10:21.107469 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:10:21.107470 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:10:21.107481 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:10:21.107552 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:10:21.107946 (+ 394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:10:21.108765 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:10:21.109393 (+ 628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:10:21.109652 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:10:21.109721 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:10:21.111643 (+ 1922us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 21:10:21.111649 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 21:10:21.111649 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 21:10:21.111655 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 21:10:21.113021 (+ 1366us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:10:21.113328 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:10:21.113330 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:10:21.113330 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:10:21.113382 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:10:21.113538 (+ 156us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:10:21.113539 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:10:21.113539 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:10:21.113668 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 21:10:21.113734 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0924 21:10:21.113887 (+ 153us) server_negotiation.cc:299] Negotiation successful
0924 21:10:21.113960 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100,"thread_start_us":60,"threads_started":1}
I20260924 21:10:21.114497 14772 tablet_service.cc:1514] Processing DeleteTablet for tablet 21d54bdd881a4d4888e1a1acc512bd14 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 21:10:20 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:57164
I20260924 21:10:21.114941 14846 tablet_replica.cc:330] stopping tablet replica
I20260924 21:10:21.115038 14846 raft_consensus.cc:2229] T 21d54bdd881a4d4888e1a1acc512bd14 P 98dbb90780044a349f025e385e51e5c7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 21:10:21.115093 14846 raft_consensus.cc:2258] T 21d54bdd881a4d4888e1a1acc512bd14 P 98dbb90780044a349f025e385e51e5c7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 21:10:21.115307 14846 ts_tablet_manager.cc:1905] T 21d54bdd881a4d4888e1a1acc512bd14 P 98dbb90780044a349f025e385e51e5c7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 21:10:21.117169 14846 ts_tablet_manager.cc:1918] T 21d54bdd881a4d4888e1a1acc512bd14 P 98dbb90780044a349f025e385e51e5c7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 21:10:21.117264 14846 log.cc:1196] T 21d54bdd881a4d4888e1a1acc512bd14 P 98dbb90780044a349f025e385e51e5c7: Deleting WAL directory at /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790284076800039-392-0/minicluster-data/ts-0/wal/wals/21d54bdd881a4d4888e1a1acc512bd14
I20260924 21:10:21.117480 14846 ts_tablet_manager.cc:1939] T 21d54bdd881a4d4888e1a1acc512bd14 P 98dbb90780044a349f025e385e51e5c7: Deleting consensus metadata
I20260924 21:10:21.117972 14636 catalog_manager.cc:4841] TS 98dbb90780044a349f025e385e51e5c7 (127.0.98.1:44771): tablet 21d54bdd881a4d4888e1a1acc512bd14 (table test-table [id=7dc021814272415dbe159263a5ae9e75]) successfully deleted
W20260924 21:10:21.176280 14843 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 21:10:21.176465 14843 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 21:10:21.176499 14843 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 21:10:21.177479 14843 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 21:10:21.177532 14843 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 21:10:21.177570 14843 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 21:10:21.178889 14843 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35231
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790284076800039-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790284076800039-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790284076800039-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:41909
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790284076800039-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=46117
--tserver_master_addrs=127.0.98.62:45663
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14843
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790284076800039-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 19:49:06 UTC on ccycloud.quasar-vhqjoc.root.comops.site
build id 4670864
I20260924 21:10:21.179116 14843 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 21:10:21.179322 14843 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 21:10:21.183703 14851 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 21:10:21.183700 14854 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 21:10:21.183701 14852 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 21:10:21.183830 14843 server_base.cc:1034] running on GCE node
I20260924 21:10:21.184012 14843 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 21:10:21.184229 14843 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 21:10:21.185457 14843 hybrid_clock.cc:648] HybridClock initialized: now 1790284221185440 us; error 31 us; skew 500 ppm
Sep 24 21:10:21 dist-test-slave-c7tk krb5kdc[14071](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284221, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 21:10:21.187393 14843 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 21:10:21.187752 14843 webserver.cc:460] Webserver started at http://127.0.98.2:46117/ using document root <none> and password file <none>
I20260924 21:10:21.187902 14843 fs_manager.cc:359] Metadata directory not provided
I20260924 21:10:21.187945 14843 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 21:10:21.188921 14843 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.001s sys 0.000s
I20260924 21:10:21.189393 14861 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 21:10:21.189534 14843 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 21:10:21.189610 14843 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790284076800039-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790284076800039-392-0/minicluster-data/ts-1/wal
uuid: "6125c714aa5640fba1a78fb9c555e3d2"
format_stamp: "Formatted at 2026-09-24 21:10:20 on dist-test-slave-c7tk"
server_key: "285202fe52dbb23bcbcc83d50a1c0497"
server_key_iv: "dfc07fb97f92866d307feec1d056a08e"
server_key_version: "encryptionkey@0"
I20260924 21:10:21.189903 14843 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790284076800039-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790284076800039-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790284076800039-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 21:10:21.211681 14843 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 21:10:21.213192 14843 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 21:10:21.213336 14843 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 21:10:21.213534 14843 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 21:10:21.213961 14868 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
W20260924 21:10:21.215085 14868 ts_tablet_manager.cc:1852] T 21d54bdd881a4d4888e1a1acc512bd14 P 6125c714aa5640fba1a78fb9c555e3d2: Tablet Manager startup: Rolling forward tablet deletion of type TABLET_DATA_DELETED
I20260924 21:10:21.215278 14868 ts_tablet_manager.cc:1905] T 21d54bdd881a4d4888e1a1acc512bd14 P 6125c714aa5640fba1a78fb9c555e3d2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 21:10:21.217243 14868 ts_tablet_manager.cc:1918] T 21d54bdd881a4d4888e1a1acc512bd14 P 6125c714aa5640fba1a78fb9c555e3d2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 21:10:21.217304 14868 log.cc:1196] T 21d54bdd881a4d4888e1a1acc512bd14 P 6125c714aa5640fba1a78fb9c555e3d2: Deleting WAL directory at /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790284076800039-392-0/minicluster-data/ts-1/wal/wals/21d54bdd881a4d4888e1a1acc512bd14
I20260924 21:10:21.217568 14868 ts_tablet_manager.cc:1939] T 21d54bdd881a4d4888e1a1acc512bd14 P 6125c714aa5640fba1a78fb9c555e3d2: Deleting consensus metadata
I20260924 21:10:21.217880 14843 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 0 live tablets)
I20260924 21:10:21.217940 14843 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.004s user 0.000s sys 0.000s
I20260924 21:10:21.217978 14843 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 21:10:21.218003 14843 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 21:10:21.223280 14843 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:41909
I20260924 21:10:21.223327 14975 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:41909 every 8 connection(s)
I20260924 21:10:21.223553 14843 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790284076800039-392-0/minicluster-data/ts-1/data/info.pb
I20260924 21:10:21.226536 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAzNNjS/build/release/bin/kudu as pid 14843
I20260924 21:10:21.226783 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAzNNjS/build/release/bin/kudu
/tmp/dist-test-taskAzNNjS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790284076800039-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790284076800039-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790284076800039-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790284076800039-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:46821
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:45663
--webserver_port=40127
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790284076800039-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35231
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790284076800039-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790284076800039-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790284076800039-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 21:10:21 dist-test-slave-c7tk krb5kdc[14071](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284221, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 21:10:21.231614 14841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:10:21.224113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:41563 (local address 127.0.98.62:45663)
0924 21:10:21.224156 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0924 21:10:21.224158 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 21:10:21.224365 (+ 207us) server_negotiation.cc:407] Connection header received
0924 21:10:21.224716 (+ 351us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:10:21.224717 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:10:21.224726 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:10:21.224771 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:10:21.225224 (+ 453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:10:21.225661 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:10:21.226706 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:10:21.226903 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:10:21.226968 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:10:21.229043 (+ 2075us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 21:10:21.229044 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 21:10:21.229045 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 21:10:21.229050 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 21:10:21.230223 (+ 1173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:10:21.230704 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:10:21.230705 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:10:21.230705 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:10:21.230749 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:10:21.231028 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:10:21.231028 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:10:21.231029 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:10:21.231120 (+ 91us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 21:10:21.231167 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0924 21:10:21.231486 (+ 319us) server_negotiation.cc:299] Negotiation successful
0924 21:10:21.231559 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260924 21:10:21.231890 14978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:10:21.224123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45663 (local address 127.0.98.2:41563)
0924 21:10:21.224269 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0924 21:10:21.224296 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0924 21:10:21.224649 (+ 353us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 21:10:21.224823 (+ 174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 21:10:21.224825 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 21:10:21.225016 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0924 21:10:21.225163 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:10:21.225166 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:10:21.225728 (+ 562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:10:21.225729 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:10:21.226599 (+ 870us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:10:21.226603 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:10:21.227012 (+ 409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:10:21.227013 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:10:21.227141 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:10:21.227556 (+ 415us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 21:10:21.227564 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 21:10:21.228941 (+ 1377us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 21:10:21.230290 (+ 1349us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 21:10:21.230296 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 21:10:21.230298 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 21:10:21.230628 (+ 330us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 21:10:21.230816 (+ 188us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 21:10:21.230817 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 21:10:21.230817 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 21:10:21.230964 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 21:10:21.231182 (+ 218us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 21:10:21.231188 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 21:10:21.231349 (+ 161us) client_negotiation.cc:769] Sending connection context
0924 21:10:21.231470 (+ 121us) client_negotiation.cc:240] Negotiation successful
0924 21:10:21.231645 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":77,"threads_started":1}
I20260924 21:10:21.232309 14976 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45663
I20260924 21:10:21.232388 14976 heartbeater.cc:464] Registering TS with master...
I20260924 21:10:21.232542 14976 heartbeater.cc:511] Master 127.0.98.62:45663 requested a full tablet report, sending...
I20260924 21:10:21.232849 14649 ts_manager.cc:194] Registered new tserver with Master: 6125c714aa5640fba1a78fb9c555e3d2 (127.0.98.2:41909)
I20260924 21:10:21.233148 14649 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:41563
W20260924 21:10:21.300215 14979 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 21:10:21.300450 14979 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 21:10:21.300488 14979 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 21:10:21.301662 14979 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 21:10:21.301716 14979 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 21:10:21.301754 14979 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 21:10:21.303069 14979 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35231
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790284076800039-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790284076800039-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790284076800039-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:46821
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790284076800039-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=40127
--tserver_master_addrs=127.0.98.62:45663
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14979
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790284076800039-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 19:49:06 UTC on ccycloud.quasar-vhqjoc.root.comops.site
build id 4670864
I20260924 21:10:21.303333 14979 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 21:10:21.303520 14979 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 21:10:21.307607 14987 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 21:10:21.307577 14985 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 21:10:21.307581 14984 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 21:10:21.307803 14979 server_base.cc:1034] running on GCE node
I20260924 21:10:21.308004 14979 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 21:10:21.308207 14979 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 21:10:21.309351 14979 hybrid_clock.cc:648] HybridClock initialized: now 1790284221309327 us; error 33 us; skew 500 ppm
Sep 24 21:10:21 dist-test-slave-c7tk krb5kdc[14071](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284221, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 21:10:21.311560 14979 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 21:10:21.311911 14979 webserver.cc:460] Webserver started at http://127.0.98.3:40127/ using document root <none> and password file <none>
I20260924 21:10:21.312095 14979 fs_manager.cc:359] Metadata directory not provided
I20260924 21:10:21.312163 14979 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 21:10:21.313248 14979 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 21:10:21.313776 14994 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 21:10:21.313956 14979 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 21:10:21.314033 14979 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790284076800039-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790284076800039-392-0/minicluster-data/ts-2/wal
uuid: "985271ad73854c8eb773b3da78a409dc"
format_stamp: "Formatted at 2026-09-24 21:10:20 on dist-test-slave-c7tk"
server_key: "08f89c81555537cd2f66dbbeaf5bf73e"
server_key_iv: "e72cffdb59c19b9a104557b64137e840"
server_key_version: "encryptionkey@0"
I20260924 21:10:21.314263 14979 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790284076800039-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790284076800039-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790284076800039-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 21:10:21.336468 14979 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 21:10:21.338142 14979 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 21:10:21.338232 14979 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 21:10:21.338367 14979 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 21:10:21.338657 14979 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 21:10:21.338702 14979 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 21:10:21.338757 14979 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 21:10:21.338785 14979 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 21:10:21.343921 14979 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:46821
I20260924 21:10:21.343953 15107 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:46821 every 8 connection(s)
I20260924 21:10:21.344218 14979 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790284076800039-392-0/minicluster-data/ts-2/data/info.pb
Sep 24 21:10:21 dist-test-slave-c7tk krb5kdc[14071](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284221, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 21:10:21.351399 14841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:10:21.344815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:50445 (local address 127.0.98.62:45663)
0924 21:10:21.344856 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0924 21:10:21.344857 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 21:10:21.345005 (+ 148us) server_negotiation.cc:407] Connection header received
0924 21:10:21.345332 (+ 327us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:10:21.345332 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:10:21.345341 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:10:21.345394 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:10:21.346145 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:10:21.346550 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:10:21.347360 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:10:21.347521 (+ 161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:10:21.347563 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:10:21.349137 (+ 1574us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 21:10:21.349138 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 21:10:21.349138 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 21:10:21.349143 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 21:10:21.350379 (+ 1236us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:10:21.350703 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:10:21.350704 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:10:21.350704 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:10:21.350749 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:10:21.350939 (+ 190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:10:21.350939 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:10:21.350940 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:10:21.351030 (+ 90us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 21:10:21.351077 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0924 21:10:21.351276 (+ 199us) server_negotiation.cc:299] Negotiation successful
0924 21:10:21.351333 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260924 21:10:21.351562 15110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:10:21.344802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45663 (local address 127.0.98.3:50445)
0924 21:10:21.344914 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0924 21:10:21.344940 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0924 21:10:21.345265 (+ 325us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 21:10:21.345435 (+ 170us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 21:10:21.345438 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 21:10:21.345933 (+ 495us) client_negotiation.cc:189] Negotiated authn=SASL
0924 21:10:21.346079 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:10:21.346083 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:10:21.346632 (+ 549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:10:21.346632 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:10:21.347291 (+ 659us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:10:21.347293 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:10:21.347580 (+ 287us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:10:21.347580 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:10:21.347658 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:10:21.347978 (+ 320us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 21:10:21.347984 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0924 21:10:21.349063 (+ 1079us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 21:10:21.350448 (+ 1385us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 21:10:21.350453 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 21:10:21.350454 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 21:10:21.350641 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 21:10:21.350803 (+ 162us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 21:10:21.350803 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 21:10:21.350803 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 21:10:21.350886 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 21:10:21.351080 (+ 194us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 21:10:21.351083 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 21:10:21.351176 (+ 93us) client_negotiation.cc:769] Sending connection context
0924 21:10:21.351249 (+ 73us) client_negotiation.cc:240] Negotiation successful
0924 21:10:21.351360 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":85,"thread_start_us":67,"threads_started":1}
I20260924 21:10:21.351655 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAzNNjS/build/release/bin/kudu as pid 14979
I20260924 21:10:21.351974 15108 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45663
I20260924 21:10:21.352048 15108 heartbeater.cc:464] Registering TS with master...
I20260924 21:10:21.352201 15108 heartbeater.cc:511] Master 127.0.98.62:45663 requested a full tablet report, sending...
I20260924 21:10:21.352543 14649 ts_manager.cc:194] Registered new tserver with Master: 985271ad73854c8eb773b3da78a409dc (127.0.98.3:46821)
I20260924 21:10:21.352847 14649 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:50445
I20260924 21:10:21.357069 14841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:10:21.351982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47426 (local address 127.0.98.62:45663)
0924 21:10:21.352014 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0924 21:10:21.352016 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 21:10:21.352139 (+ 123us) server_negotiation.cc:407] Connection header received
0924 21:10:21.352195 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:10:21.352195 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:10:21.352202 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:10:21.352255 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:10:21.352863 (+ 608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:10:21.353322 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:10:21.354083 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:10:21.354278 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:10:21.354352 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:10:21.355241 (+ 889us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 21:10:21.355242 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 21:10:21.355242 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 21:10:21.355248 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 21:10:21.356131 (+ 883us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:10:21.356441 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:10:21.356442 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:10:21.356442 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:10:21.356491 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:10:21.356659 (+ 168us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:10:21.356660 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:10:21.356660 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:10:21.356746 (+ 86us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 21:10:21.356904 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0924 21:10:21.356934 (+ 30us) server_negotiation.cc:299] Negotiation successful
0924 21:10:21.356990 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260924 21:10:21.357668 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 21:10:21.359809 14841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:10:21.357980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47428 (local address 127.0.98.62:45663)
0924 21:10:21.358009 (+ 29us) server_negotiation.cc:206] Beginning negotiation
0924 21:10:21.358010 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 21:10:21.358071 (+ 61us) server_negotiation.cc:407] Connection header received
0924 21:10:21.358113 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:10:21.358113 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:10:21.358133 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:10:21.358183 (+ 50us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 21:10:21.358242 (+ 59us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:10:21.358661 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:10:21.359294 (+ 633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:10:21.359459 (+ 165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:10:21.359516 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:10:21.359634 (+ 118us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 21:10:21.359697 (+ 63us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 21:10:21.359739 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0924 21:10:21.359766 (+ 27us) server_negotiation.cc:299] Negotiation successful
0924 21:10:21.359776 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14}
I20260924 21:10:21.360033 14649 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:47428:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 21:10:21.361526 14649 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 21:10:21.363251 14772 tablet_service.cc:1467] Processing CreateTablet for tablet 44208eb9cbaa49269197c60f035e4d02 (DEFAULT_TABLE table=test-table [id=b406998932a34b41871e8c0b81384b22]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 21:10:21.363416 14772 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 44208eb9cbaa49269197c60f035e4d02. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 21:10:21.364693 14729 tablet_bootstrap.cc:492] T 44208eb9cbaa49269197c60f035e4d02 P 98dbb90780044a349f025e385e51e5c7: Bootstrap starting.
I20260924 21:10:21.365058 14729 tablet_bootstrap.cc:654] T 44208eb9cbaa49269197c60f035e4d02 P 98dbb90780044a349f025e385e51e5c7: Neither blocks nor log segments found. Creating new log.
I20260924 21:10:21.365770 14729 tablet_bootstrap.cc:492] T 44208eb9cbaa49269197c60f035e4d02 P 98dbb90780044a349f025e385e51e5c7: No bootstrap required, opened a new log
I20260924 21:10:21.365832 14729 ts_tablet_manager.cc:1397] T 44208eb9cbaa49269197c60f035e4d02 P 98dbb90780044a349f025e385e51e5c7: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260924 21:10:21.365974 14729 raft_consensus.cc:348] T 44208eb9cbaa49269197c60f035e4d02 P 98dbb90780044a349f025e385e51e5c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "985271ad73854c8eb773b3da78a409dc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46821 } } peers { permanent_uuid: "6125c714aa5640fba1a78fb9c555e3d2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41909 } } peers { permanent_uuid: "98dbb90780044a349f025e385e51e5c7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44771 } }
I20260924 21:10:21.366042 14729 raft_consensus.cc:374] T 44208eb9cbaa49269197c60f035e4d02 P 98dbb90780044a349f025e385e51e5c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 21:10:21.366088 14729 raft_consensus.cc:729] T 44208eb9cbaa49269197c60f035e4d02 P 98dbb90780044a349f025e385e51e5c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98dbb90780044a349f025e385e51e5c7, State: Initialized, Role: FOLLOWER
I20260924 21:10:21.366127 14729 consensus_queue.cc:260] T 44208eb9cbaa49269197c60f035e4d02 P 98dbb90780044a349f025e385e51e5c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "985271ad73854c8eb773b3da78a409dc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46821 } } peers { permanent_uuid: "6125c714aa5640fba1a78fb9c555e3d2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41909 } } peers { permanent_uuid: "98dbb90780044a349f025e385e51e5c7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44771 } }
I20260924 21:10:21.366207 14729 ts_tablet_manager.cc:1428] T 44208eb9cbaa49269197c60f035e4d02 P 98dbb90780044a349f025e385e51e5c7: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 21:10:21.366338 15113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:10:21.363022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41909 (local address 127.0.0.1:55548)
0924 21:10:21.363208 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0924 21:10:21.363217 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 21:10:21.363272 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 21:10:21.363567 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 21:10:21.363568 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 21:10:21.363570 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 21:10:21.363606 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:10:21.363608 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:10:21.364647 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:10:21.364648 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:10:21.365291 (+ 643us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:10:21.365293 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:10:21.366119 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:10:21.366119 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:10:21.366180 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:10:21.366211 (+ 31us) client_negotiation.cc:769] Sending connection context
0924 21:10:21.366242 (+ 31us) client_negotiation.cc:240] Negotiation successful
0924 21:10:21.366281 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":42,"threads_started":1}
I20260924 21:10:21.366550 14844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:10:21.362975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:46821 (local address 127.0.0.1:37554)
0924 21:10:21.363001 (+ 26us) negotiation.cc:107] Waiting for socket to connect
0924 21:10:21.363009 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0924 21:10:21.363041 (+ 32us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 21:10:21.363530 (+ 489us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 21:10:21.363531 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 21:10:21.363532 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 21:10:21.363569 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:10:21.363570 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:10:21.364609 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:10:21.364610 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:10:21.365583 (+ 973us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:10:21.365585 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:10:21.366366 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:10:21.366366 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:10:21.366427 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:10:21.366430 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 21:10:21.366468 (+ 38us) client_negotiation.cc:240] Negotiation successful
0924 21:10:21.366499 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14}
I20260924 21:10:21.366639 15115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:10:21.363227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55548 (local address 127.0.98.2:41909)
0924 21:10:21.363364 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0924 21:10:21.363368 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 21:10:21.363379 (+ 11us) server_negotiation.cc:407] Connection header received
0924 21:10:21.363415 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:10:21.363417 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:10:21.363433 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:10:21.363505 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 21:10:21.363676 (+ 171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:10:21.364575 (+ 899us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:10:21.365392 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:10:21.366030 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:10:21.366450 (+ 420us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:10:21.366506 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0924 21:10:21.366527 (+ 21us) server_negotiation.cc:299] Negotiation successful
0924 21:10:21.366559 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113,"thread_start_us":89,"threads_started":1}
I20260924 21:10:21.366724 15114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:10:21.363068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37554 (local address 127.0.98.3:46821)
0924 21:10:21.363412 (+ 344us) server_negotiation.cc:206] Beginning negotiation
0924 21:10:21.363415 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 21:10:21.363422 (+ 7us) server_negotiation.cc:407] Connection header received
0924 21:10:21.363446 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:10:21.363447 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:10:21.363461 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:10:21.363521 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 21:10:21.363623 (+ 102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:10:21.364510 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:10:21.365660 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:10:21.366235 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:10:21.366284 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:10:21.366326 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0924 21:10:21.366629 (+ 303us) server_negotiation.cc:299] Negotiation successful
0924 21:10:21.366664 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":322,"thread_start_us":75,"threads_started":1}
I20260924 21:10:21.367121 14910 tablet_service.cc:1467] Processing CreateTablet for tablet 44208eb9cbaa49269197c60f035e4d02 (DEFAULT_TABLE table=test-table [id=b406998932a34b41871e8c0b81384b22]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 21:10:21.367349 14910 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 44208eb9cbaa49269197c60f035e4d02. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 21:10:21.368708 14868 tablet_bootstrap.cc:492] T 44208eb9cbaa49269197c60f035e4d02 P 6125c714aa5640fba1a78fb9c555e3d2: Bootstrap starting.
I20260924 21:10:21.369020 15042 tablet_service.cc:1467] Processing CreateTablet for tablet 44208eb9cbaa49269197c60f035e4d02 (DEFAULT_TABLE table=test-table [id=b406998932a34b41871e8c0b81384b22]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 21:10:21.369283 15042 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 44208eb9cbaa49269197c60f035e4d02. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 21:10:21.369405 14868 tablet_bootstrap.cc:654] T 44208eb9cbaa49269197c60f035e4d02 P 6125c714aa5640fba1a78fb9c555e3d2: Neither blocks nor log segments found. Creating new log.
I20260924 21:10:21.369705 14868 log.cc:824] T 44208eb9cbaa49269197c60f035e4d02 P 6125c714aa5640fba1a78fb9c555e3d2: Log is configured to *not* fsync() on all Append() calls
I20260924 21:10:21.370292 14868 tablet_bootstrap.cc:492] T 44208eb9cbaa49269197c60f035e4d02 P 6125c714aa5640fba1a78fb9c555e3d2: No bootstrap required, opened a new log
I20260924 21:10:21.370385 14868 ts_tablet_manager.cc:1397] T 44208eb9cbaa49269197c60f035e4d02 P 6125c714aa5640fba1a78fb9c555e3d2: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 21:10:21.371639 14868 raft_consensus.cc:348] T 44208eb9cbaa49269197c60f035e4d02 P 6125c714aa5640fba1a78fb9c555e3d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "985271ad73854c8eb773b3da78a409dc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46821 } } peers { permanent_uuid: "6125c714aa5640fba1a78fb9c555e3d2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41909 } } peers { permanent_uuid: "98dbb90780044a349f025e385e51e5c7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44771 } }
I20260924 21:10:21.371759 15118 tablet_bootstrap.cc:492] T 44208eb9cbaa49269197c60f035e4d02 P 985271ad73854c8eb773b3da78a409dc: Bootstrap starting.
I20260924 21:10:21.371809 14868 raft_consensus.cc:374] T 44208eb9cbaa49269197c60f035e4d02 P 6125c714aa5640fba1a78fb9c555e3d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 21:10:21.371848 14868 raft_consensus.cc:729] T 44208eb9cbaa49269197c60f035e4d02 P 6125c714aa5640fba1a78fb9c555e3d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6125c714aa5640fba1a78fb9c555e3d2, State: Initialized, Role: FOLLOWER
I20260924 21:10:21.371987 14868 consensus_queue.cc:260] T 44208eb9cbaa49269197c60f035e4d02 P 6125c714aa5640fba1a78fb9c555e3d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "985271ad73854c8eb773b3da78a409dc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46821 } } peers { permanent_uuid: "6125c714aa5640fba1a78fb9c555e3d2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41909 } } peers { permanent_uuid: "98dbb90780044a349f025e385e51e5c7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44771 } }
I20260924 21:10:21.372262 14976 heartbeater.cc:503] Master 127.0.98.62:45663 was elected leader, sending a full tablet report...
I20260924 21:10:21.372247 14868 ts_tablet_manager.cc:1428] T 44208eb9cbaa49269197c60f035e4d02 P 6125c714aa5640fba1a78fb9c555e3d2: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 21:10:21.372375 15118 tablet_bootstrap.cc:654] T 44208eb9cbaa49269197c60f035e4d02 P 985271ad73854c8eb773b3da78a409dc: Neither blocks nor log segments found. Creating new log.
I20260924 21:10:21.372829 15118 log.cc:824] T 44208eb9cbaa49269197c60f035e4d02 P 985271ad73854c8eb773b3da78a409dc: Log is configured to *not* fsync() on all Append() calls
I20260924 21:10:21.373490 15118 tablet_bootstrap.cc:492] T 44208eb9cbaa49269197c60f035e4d02 P 985271ad73854c8eb773b3da78a409dc: No bootstrap required, opened a new log
I20260924 21:10:21.373535 15118 ts_tablet_manager.cc:1397] T 44208eb9cbaa49269197c60f035e4d02 P 985271ad73854c8eb773b3da78a409dc: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 21:10:21.374665 15118 raft_consensus.cc:348] T 44208eb9cbaa49269197c60f035e4d02 P 985271ad73854c8eb773b3da78a409dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "985271ad73854c8eb773b3da78a409dc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46821 } } peers { permanent_uuid: "6125c714aa5640fba1a78fb9c555e3d2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41909 } } peers { permanent_uuid: "98dbb90780044a349f025e385e51e5c7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44771 } }
I20260924 21:10:21.374773 15118 raft_consensus.cc:374] T 44208eb9cbaa49269197c60f035e4d02 P 985271ad73854c8eb773b3da78a409dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 21:10:21.374790 15118 raft_consensus.cc:729] T 44208eb9cbaa49269197c60f035e4d02 P 985271ad73854c8eb773b3da78a409dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 985271ad73854c8eb773b3da78a409dc, State: Initialized, Role: FOLLOWER
I20260924 21:10:21.374850 15118 consensus_queue.cc:260] T 44208eb9cbaa49269197c60f035e4d02 P 985271ad73854c8eb773b3da78a409dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "985271ad73854c8eb773b3da78a409dc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46821 } } peers { permanent_uuid: "6125c714aa5640fba1a78fb9c555e3d2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41909 } } peers { permanent_uuid: "98dbb90780044a349f025e385e51e5c7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44771 } }
I20260924 21:10:21.375023 15118 ts_tablet_manager.cc:1428] T 44208eb9cbaa49269197c60f035e4d02 P 985271ad73854c8eb773b3da78a409dc: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260924 21:10:21.375062 15108 heartbeater.cc:503] Master 127.0.98.62:45663 was elected leader, sending a full tablet report...
I20260924 21:10:21.466434 15119 raft_consensus.cc:482] T 44208eb9cbaa49269197c60f035e4d02 P 6125c714aa5640fba1a78fb9c555e3d2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 21:10:21.466575 15119 raft_consensus.cc:504] T 44208eb9cbaa49269197c60f035e4d02 P 6125c714aa5640fba1a78fb9c555e3d2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "985271ad73854c8eb773b3da78a409dc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46821 } } peers { permanent_uuid: "6125c714aa5640fba1a78fb9c555e3d2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41909 } } peers { permanent_uuid: "98dbb90780044a349f025e385e51e5c7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44771 } }
I20260924 21:10:21.466917 15119 leader_election.cc:288] T 44208eb9cbaa49269197c60f035e4d02 P 6125c714aa5640fba1a78fb9c555e3d2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 985271ad73854c8eb773b3da78a409dc (127.0.98.3:46821), 98dbb90780044a349f025e385e51e5c7 (127.0.98.1:44771)
I20260924 21:10:21.469307 14978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:10:21.467058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:46821 (local address 127.0.98.2:44489)
0924 21:10:21.467097 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0924 21:10:21.467106 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 21:10:21.467141 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 21:10:21.467268 (+ 127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 21:10:21.467269 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 21:10:21.467270 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 21:10:21.467332 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:10:21.467334 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:10:21.467868 (+ 534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:10:21.467869 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:10:21.468582 (+ 713us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:10:21.468584 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:10:21.469089 (+ 505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:10:21.469090 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:10:21.469161 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:10:21.469163 (+ 2us) client_negotiation.cc:769] Sending connection context
0924 21:10:21.469215 (+ 52us) client_negotiation.cc:240] Negotiation successful
0924 21:10:21.469251 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":22}
I20260924 21:10:21.469336 15114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:10:21.467081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:44489 (local address 127.0.98.3:46821)
0924 21:10:21.467121 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0924 21:10:21.467122 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 21:10:21.467131 (+ 9us) server_negotiation.cc:407] Connection header received
0924 21:10:21.467203 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:10:21.467204 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:10:21.467213 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:10:21.467263 (+ 50us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 21:10:21.467395 (+ 132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:10:21.467785 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:10:21.468667 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:10:21.469023 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:10:21.469090 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:10:21.469098 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0924 21:10:21.469232 (+ 134us) server_negotiation.cc:299] Negotiation successful
0924 21:10:21.469279 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260924 21:10:21.469588 15062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44208eb9cbaa49269197c60f035e4d02" candidate_uuid: "6125c714aa5640fba1a78fb9c555e3d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "985271ad73854c8eb773b3da78a409dc" is_pre_election: true
I20260924 21:10:21.469725 15062 raft_consensus.cc:2454] T 44208eb9cbaa49269197c60f035e4d02 P 985271ad73854c8eb773b3da78a409dc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6125c714aa5640fba1a78fb9c555e3d2 in term 0.
I20260924 21:10:21.469944 14863 leader_election.cc:302] T 44208eb9cbaa49269197c60f035e4d02 P 6125c714aa5640fba1a78fb9c555e3d2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6125c714aa5640fba1a78fb9c555e3d2, 985271ad73854c8eb773b3da78a409dc; no voters:
I20260924 21:10:21.470063 15119 raft_consensus.cc:2781] T 44208eb9cbaa49269197c60f035e4d02 P 6125c714aa5640fba1a78fb9c555e3d2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 21:10:21.470104 15119 raft_consensus.cc:482] T 44208eb9cbaa49269197c60f035e4d02 P 6125c714aa5640fba1a78fb9c555e3d2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 21:10:21.470139 15119 raft_consensus.cc:3037] T 44208eb9cbaa49269197c60f035e4d02 P 6125c714aa5640fba1a78fb9c555e3d2 [term 0 FOLLOWER]: Advancing to term 1
I20260924 21:10:21.470186 14845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:10:21.467129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:39459 (local address 127.0.98.1:44771)
0924 21:10:21.467190 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0924 21:10:21.467192 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 21:10:21.467399 (+ 207us) server_negotiation.cc:407] Connection header received
0924 21:10:21.467449 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:10:21.467450 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:10:21.467476 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:10:21.467539 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 21:10:21.467664 (+ 125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:10:21.468075 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:10:21.469249 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:10:21.469835 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:10:21.469920 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:10:21.469993 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0924 21:10:21.470061 (+ 68us) server_negotiation.cc:299] Negotiation successful
0924 21:10:21.470099 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260924 21:10:21.470278 15122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:10:21.467064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44771 (local address 127.0.98.2:39459)
0924 21:10:21.467345 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0924 21:10:21.467353 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0924 21:10:21.467401 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 21:10:21.467541 (+ 140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 21:10:21.467541 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 21:10:21.467543 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 21:10:21.467602 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:10:21.467604 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:10:21.468212 (+ 608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:10:21.468213 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:10:21.469049 (+ 836us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:10:21.469052 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:10:21.469932 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:10:21.469934 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:10:21.469994 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:10:21.469998 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 21:10:21.470055 (+ 57us) client_negotiation.cc:240] Negotiation successful
0924 21:10:21.470088 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"mutex_wait_us":137,"thread_start_us":102,"threads_started":1}
I20260924 21:10:21.470650 14792 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44208eb9cbaa49269197c60f035e4d02" candidate_uuid: "6125c714aa5640fba1a78fb9c555e3d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98dbb90780044a349f025e385e51e5c7" is_pre_election: true
I20260924 21:10:21.470798 14792 raft_consensus.cc:2454] T 44208eb9cbaa49269197c60f035e4d02 P 98dbb90780044a349f025e385e51e5c7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6125c714aa5640fba1a78fb9c555e3d2 in term 0.
I20260924 21:10:21.471022 15119 raft_consensus.cc:504] T 44208eb9cbaa49269197c60f035e4d02 P 6125c714aa5640fba1a78fb9c555e3d2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "985271ad73854c8eb773b3da78a409dc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46821 } } peers { permanent_uuid: "6125c714aa5640fba1a78fb9c555e3d2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41909 } } peers { permanent_uuid: "98dbb90780044a349f025e385e51e5c7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44771 } }
I20260924 21:10:21.471159 15119 leader_election.cc:288] T 44208eb9cbaa49269197c60f035e4d02 P 6125c714aa5640fba1a78fb9c555e3d2 [CANDIDATE]: Term 1 election: Requested vote from peers 985271ad73854c8eb773b3da78a409dc (127.0.98.3:46821), 98dbb90780044a349f025e385e51e5c7 (127.0.98.1:44771)
I20260924 21:10:21.471297 15062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44208eb9cbaa49269197c60f035e4d02" candidate_uuid: "6125c714aa5640fba1a78fb9c555e3d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "985271ad73854c8eb773b3da78a409dc"
I20260924 21:10:21.471297 14792 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44208eb9cbaa49269197c60f035e4d02" candidate_uuid: "6125c714aa5640fba1a78fb9c555e3d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98dbb90780044a349f025e385e51e5c7"
I20260924 21:10:21.471375 14792 raft_consensus.cc:3037] T 44208eb9cbaa49269197c60f035e4d02 P 98dbb90780044a349f025e385e51e5c7 [term 0 FOLLOWER]: Advancing to term 1
I20260924 21:10:21.471386 15062 raft_consensus.cc:3037] T 44208eb9cbaa49269197c60f035e4d02 P 985271ad73854c8eb773b3da78a409dc [term 0 FOLLOWER]: Advancing to term 1
I20260924 21:10:21.472148 14792 raft_consensus.cc:2454] T 44208eb9cbaa49269197c60f035e4d02 P 98dbb90780044a349f025e385e51e5c7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6125c714aa5640fba1a78fb9c555e3d2 in term 1.
I20260924 21:10:21.472147 15062 raft_consensus.cc:2454] T 44208eb9cbaa49269197c60f035e4d02 P 985271ad73854c8eb773b3da78a409dc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6125c714aa5640fba1a78fb9c555e3d2 in term 1.
I20260924 21:10:21.472304 14865 leader_election.cc:302] T 44208eb9cbaa49269197c60f035e4d02 P 6125c714aa5640fba1a78fb9c555e3d2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6125c714aa5640fba1a78fb9c555e3d2, 98dbb90780044a349f025e385e51e5c7; no voters:
I20260924 21:10:21.472395 15119 raft_consensus.cc:2781] T 44208eb9cbaa49269197c60f035e4d02 P 6125c714aa5640fba1a78fb9c555e3d2 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 21:10:21.472512 15119 raft_consensus.cc:686] T 44208eb9cbaa49269197c60f035e4d02 P 6125c714aa5640fba1a78fb9c555e3d2 [term 1 LEADER]: Becoming Leader. State: Replica: 6125c714aa5640fba1a78fb9c555e3d2, State: Running, Role: LEADER
I20260924 21:10:21.472599 15119 consensus_queue.cc:237] T 44208eb9cbaa49269197c60f035e4d02 P 6125c714aa5640fba1a78fb9c555e3d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "985271ad73854c8eb773b3da78a409dc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46821 } } peers { permanent_uuid: "6125c714aa5640fba1a78fb9c555e3d2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41909 } } peers { permanent_uuid: "98dbb90780044a349f025e385e51e5c7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44771 } }
I20260924 21:10:21.473274 14649 catalog_manager.cc:5495] T 44208eb9cbaa49269197c60f035e4d02 P 6125c714aa5640fba1a78fb9c555e3d2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6125c714aa5640fba1a78fb9c555e3d2 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "6125c714aa5640fba1a78fb9c555e3d2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "985271ad73854c8eb773b3da78a409dc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46821 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6125c714aa5640fba1a78fb9c555e3d2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41909 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "98dbb90780044a349f025e385e51e5c7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44771 } health_report { overall_health: UNKNOWN } } }
W20260924 21:10:21.473956 14977 tablet.cc:2369] T 44208eb9cbaa49269197c60f035e4d02 P 6125c714aa5640fba1a78fb9c555e3d2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 21:10:21.481746 15115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:10:21.479712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55552 (local address 127.0.98.2:41909)
0924 21:10:21.479753 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0924 21:10:21.479755 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 21:10:21.479762 (+ 7us) server_negotiation.cc:407] Connection header received
0924 21:10:21.479790 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:10:21.479790 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:10:21.479804 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:10:21.479857 (+ 53us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 21:10:21.479943 (+ 86us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:10:21.480299 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:10:21.481013 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:10:21.481207 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:10:21.481284 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:10:21.481445 (+ 161us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 21:10:21.481536 (+ 91us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 21:10:21.481640 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0924 21:10:21.481661 (+ 21us) server_negotiation.cc:299] Negotiation successful
0924 21:10:21.481674 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260924 21:10:21.483089 15062 raft_consensus.cc:1261] T 44208eb9cbaa49269197c60f035e4d02 P 985271ad73854c8eb773b3da78a409dc [term 1 FOLLOWER]: Refusing update from remote peer 6125c714aa5640fba1a78fb9c555e3d2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 21:10:21.483121 14792 raft_consensus.cc:1261] T 44208eb9cbaa49269197c60f035e4d02 P 98dbb90780044a349f025e385e51e5c7 [term 1 FOLLOWER]: Refusing update from remote peer 6125c714aa5640fba1a78fb9c555e3d2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 21:10:21.483359 15123 consensus_queue.cc:1035] T 44208eb9cbaa49269197c60f035e4d02 P 6125c714aa5640fba1a78fb9c555e3d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "985271ad73854c8eb773b3da78a409dc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 21:10:21.483469 15119 consensus_queue.cc:1035] T 44208eb9cbaa49269197c60f035e4d02 P 6125c714aa5640fba1a78fb9c555e3d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "98dbb90780044a349f025e385e51e5c7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44771 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 21:10:21.485904 15126 mvcc.cc:205] Tried to move back new op lower bound from 7333004171192864768 to 7333004171152281600. Current Snapshot: MvccSnapshot[applied={T|T < 7333004171192864768}]
I20260924 21:10:21.487255 14648 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:47428:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 21:10:21.487325 14648 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:47428:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 21:10:21.487900 14648 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4e3cd7d6c2b4441093bfb5a89a0767ae: Sending DeleteTablet for 3 replicas of tablet 44208eb9cbaa49269197c60f035e4d02
I20260924 21:10:21.488150 15042 tablet_service.cc:1514] Processing DeleteTablet for tablet 44208eb9cbaa49269197c60f035e4d02 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 21:10:21 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:37554
I20260924 21:10:21.488209 14910 tablet_service.cc:1514] Processing DeleteTablet for tablet 44208eb9cbaa49269197c60f035e4d02 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 21:10:21 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:55548
I20260924 21:10:21.488266 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAzNNjS/build/release/bin/kudu with pid 14688
I20260924 21:10:21.488308 14772 tablet_service.cc:1514] Processing DeleteTablet for tablet 44208eb9cbaa49269197c60f035e4d02 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 21:10:21 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:57164
I20260924 21:10:21.489028 15137 tablet_replica.cc:330] T 44208eb9cbaa49269197c60f035e4d02 P 6125c714aa5640fba1a78fb9c555e3d2: stopping tablet replica
I20260924 21:10:21.489118 15137 raft_consensus.cc:2229] T 44208eb9cbaa49269197c60f035e4d02 P 6125c714aa5640fba1a78fb9c555e3d2 [term 1 LEADER]: Raft consensus shutting down.
I20260924 21:10:21.489210 15136 tablet_replica.cc:330] T 44208eb9cbaa49269197c60f035e4d02 P 985271ad73854c8eb773b3da78a409dc: stopping tablet replica
I20260924 21:10:21.489210 15137 raft_consensus.cc:2258] T 44208eb9cbaa49269197c60f035e4d02 P 6125c714aa5640fba1a78fb9c555e3d2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 21:10:21.489285 15136 raft_consensus.cc:2229] T 44208eb9cbaa49269197c60f035e4d02 P 985271ad73854c8eb773b3da78a409dc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 21:10:21.489364 15136 raft_consensus.cc:2258] T 44208eb9cbaa49269197c60f035e4d02 P 985271ad73854c8eb773b3da78a409dc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 21:10:21.489707 15136 ts_tablet_manager.cc:1905] T 44208eb9cbaa49269197c60f035e4d02 P 985271ad73854c8eb773b3da78a409dc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 21:10:21.491035 15136 ts_tablet_manager.cc:1918] T 44208eb9cbaa49269197c60f035e4d02 P 985271ad73854c8eb773b3da78a409dc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 21:10:21.491091 15136 log.cc:1196] T 44208eb9cbaa49269197c60f035e4d02 P 985271ad73854c8eb773b3da78a409dc: Deleting WAL directory at /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790284076800039-392-0/minicluster-data/ts-2/wal/wals/44208eb9cbaa49269197c60f035e4d02
I20260924 21:10:21.491392 15136 ts_tablet_manager.cc:1939] T 44208eb9cbaa49269197c60f035e4d02 P 985271ad73854c8eb773b3da78a409dc: Deleting consensus metadata
I20260924 21:10:21.491729 14634 catalog_manager.cc:4841] TS 985271ad73854c8eb773b3da78a409dc (127.0.98.3:46821): tablet 44208eb9cbaa49269197c60f035e4d02 (table test-table [id=b406998932a34b41871e8c0b81384b22]) successfully deleted
W20260924 21:10:21.492251 14636 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:44771 (error 108)
I20260924 21:10:21.492650 15137 ts_tablet_manager.cc:1905] T 44208eb9cbaa49269197c60f035e4d02 P 6125c714aa5640fba1a78fb9c555e3d2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 21:10:21.492739 14844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:10:21.492619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44771 (local address 127.0.0.1:57166)
0924 21:10:21.492675 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0924 21:10:21.492712 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:44771: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260924 21:10:21.492846 14636 catalog_manager.cc:4584] TS 98dbb90780044a349f025e385e51e5c7 (127.0.98.1:44771): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 44208eb9cbaa49269197c60f035e4d02: Network error: Client connection negotiation failed: client connection to 127.0.98.1:44771: connect: Connection refused (error 111)
I20260924 21:10:21.493547 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAzNNjS/build/release/bin/kudu with pid 14843
I20260924 21:10:21.498024 14844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:10:21.497944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41909 (local address 127.0.0.1:55566)
0924 21:10:21.497986 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0924 21:10:21.498006 (+ 20us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:41909: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260924 21:10:21.498148 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAzNNjS/build/release/bin/kudu with pid 14979
I20260924 21:10:21.502120 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAzNNjS/build/release/bin/kudu with pid 14616
I20260924 21:10:21.506983 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAzNNjS/build/release/bin/kudu
/tmp/dist-test-taskAzNNjS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790284076800039-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790284076800039-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790284076800039-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790284076800039-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45663
--webserver_interface=127.0.98.62
--webserver_port=41717
--keytab_file=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790284076800039-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35231
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45663
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790284076800039-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790284076800039-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790284076800039-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 21:10:21.576968 15139 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 21:10:21.577122 15139 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 21:10:21.577140 15139 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 21:10:21.578238 15139 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 21:10:21.578275 15139 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 21:10:21.578289 15139 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 21:10:21.578299 15139 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 21:10:21.578336 15139 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 21:10:21.579584 15139 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35231
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790284076800039-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790284076800039-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45663
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790284076800039-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45663
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790284076800039-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=41717
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15139
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790284076800039-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 19:49:06 UTC on ccycloud.quasar-vhqjoc.root.comops.site
build id 4670864
I20260924 21:10:21.579733 15139 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 21:10:21.579933 15139 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 21:10:21.584164 15147 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 21:10:21.584242 15145 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 21:10:21.584280 15144 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 21:10:21.584302 15139 server_base.cc:1034] running on GCE node
I20260924 21:10:21.584527 15139 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 21:10:21.584708 15139 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 21:10:21.585850 15139 hybrid_clock.cc:648] HybridClock initialized: now 1790284221585821 us; error 44 us; skew 500 ppm
Sep 24 21:10:21 dist-test-slave-c7tk krb5kdc[14071](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284221, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 21:10:21.591920 15139 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 21:10:21.592329 15139 webserver.cc:460] Webserver started at http://127.0.98.62:41717/ using document root <none> and password file <none>
I20260924 21:10:21.592548 15139 fs_manager.cc:359] Metadata directory not provided
I20260924 21:10:21.592589 15139 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 21:10:21.593778 15139 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 21:10:21.594458 15154 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 21:10:21.594635 15139 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 21:10:21.594713 15139 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790284076800039-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790284076800039-392-0/minicluster-data/master-0/wal
uuid: "4e3cd7d6c2b4441093bfb5a89a0767ae"
format_stamp: "Formatted at 2026-09-24 21:10:20 on dist-test-slave-c7tk"
server_key: "e7de828d96504a2c38339d0e70f025a0"
server_key_iv: "dcbce15300373d651b4546345bfa50e7"
server_key_version: "encryptionkey@0"
I20260924 21:10:21.595028 15139 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790284076800039-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790284076800039-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790284076800039-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 21:10:21.623109 15139 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 21:10:21.624934 15139 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 21:10:21.625088 15139 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 21:10:21.629444 15139 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45663
I20260924 21:10:21.629480 15206 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45663 every 8 connection(s)
I20260924 21:10:21.629905 15139 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790284076800039-392-0/minicluster-data/master-0/data/info.pb
I20260924 21:10:21.631086 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAzNNjS/build/release/bin/kudu as pid 15139
I20260924 21:10:21.631383 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAzNNjS/build/release/bin/kudu
/tmp/dist-test-taskAzNNjS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790284076800039-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790284076800039-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790284076800039-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790284076800039-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:44771
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:45663
--webserver_port=38341
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790284076800039-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35231
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790284076800039-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790284076800039-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790284076800039-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 21:10:21.631486 15207 sys_catalog.cc:263] Verifying existing consensus state
I20260924 21:10:21.632071 15207 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4e3cd7d6c2b4441093bfb5a89a0767ae: Bootstrap starting.
I20260924 21:10:21.639869 15207 log.cc:824] T 00000000000000000000000000000000 P 4e3cd7d6c2b4441093bfb5a89a0767ae: Log is configured to *not* fsync() on all Append() calls
I20260924 21:10:21.642038 15207 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4e3cd7d6c2b4441093bfb5a89a0767ae: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 21:10:21.642289 15207 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4e3cd7d6c2b4441093bfb5a89a0767ae: Bootstrap complete.
I20260924 21:10:21.644201 15207 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4e3cd7d6c2b4441093bfb5a89a0767ae [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e3cd7d6c2b4441093bfb5a89a0767ae" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45663 } }
I20260924 21:10:21.644409 15207 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4e3cd7d6c2b4441093bfb5a89a0767ae [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e3cd7d6c2b4441093bfb5a89a0767ae, State: Initialized, Role: FOLLOWER
I20260924 21:10:21.644580 15207 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4e3cd7d6c2b4441093bfb5a89a0767ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e3cd7d6c2b4441093bfb5a89a0767ae" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45663 } }
I20260924 21:10:21.644655 15207 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4e3cd7d6c2b4441093bfb5a89a0767ae [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 21:10:21.644711 15207 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4e3cd7d6c2b4441093bfb5a89a0767ae [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 21:10:21.644757 15207 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4e3cd7d6c2b4441093bfb5a89a0767ae [term 2 FOLLOWER]: Advancing to term 3
I20260924 21:10:21.645691 15207 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4e3cd7d6c2b4441093bfb5a89a0767ae [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e3cd7d6c2b4441093bfb5a89a0767ae" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45663 } }
I20260924 21:10:21.645762 15207 leader_election.cc:302] T 00000000000000000000000000000000 P 4e3cd7d6c2b4441093bfb5a89a0767ae [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4e3cd7d6c2b4441093bfb5a89a0767ae; no voters:
I20260924 21:10:21.645946 15207 leader_election.cc:288] T 00000000000000000000000000000000 P 4e3cd7d6c2b4441093bfb5a89a0767ae [CANDIDATE]: Term 3 election: Requested vote from peers
I20260924 21:10:21.645977 15211 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4e3cd7d6c2b4441093bfb5a89a0767ae [term 3 FOLLOWER]: Leader election won for term 3
I20260924 21:10:21.646078 15211 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4e3cd7d6c2b4441093bfb5a89a0767ae [term 3 LEADER]: Becoming Leader. State: Replica: 4e3cd7d6c2b4441093bfb5a89a0767ae, State: Running, Role: LEADER
I20260924 21:10:21.646154 15211 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4e3cd7d6c2b4441093bfb5a89a0767ae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e3cd7d6c2b4441093bfb5a89a0767ae" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45663 } }
I20260924 21:10:21.646178 15207 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4e3cd7d6c2b4441093bfb5a89a0767ae [sys.catalog]: configured and running, proceeding with master startup.
I20260924 21:10:21.646688 15212 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4e3cd7d6c2b4441093bfb5a89a0767ae [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "4e3cd7d6c2b4441093bfb5a89a0767ae" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e3cd7d6c2b4441093bfb5a89a0767ae" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45663 } } }
I20260924 21:10:21.646775 15212 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4e3cd7d6c2b4441093bfb5a89a0767ae [sys.catalog]: This master's current role is: LEADER
I20260924 21:10:21.647382 15213 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4e3cd7d6c2b4441093bfb5a89a0767ae [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4e3cd7d6c2b4441093bfb5a89a0767ae. Latest consensus state: current_term: 3 leader_uuid: "4e3cd7d6c2b4441093bfb5a89a0767ae" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e3cd7d6c2b4441093bfb5a89a0767ae" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45663 } } }
I20260924 21:10:21.647593 15213 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4e3cd7d6c2b4441093bfb5a89a0767ae [sys.catalog]: This master's current role is: LEADER
I20260924 21:10:21.647888 15227 catalog_manager.cc:1259] Loaded cluster ID: 29f9a19ac9f14577a5de672c45a0000e
I20260924 21:10:21.647930 15227 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 4e3cd7d6c2b4441093bfb5a89a0767ae: loading cluster ID for follower catalog manager: success
W20260924 21:10:21.648128 15227 catalog_manager.cc:1576] T 00000000000000000000000000000000 P 4e3cd7d6c2b4441093bfb5a89a0767ae: acquiring CA information for follower catalog manager: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
W20260924 21:10:21.648165 15227 catalog_manager.cc:873] Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO: failed to prepare follower catalog manager, will retry
I20260924 21:10:21.648200 15220 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 21:10:21.648492 15220 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 21:10:21.648578 15220 catalog_manager.cc:1259] Loaded cluster ID: 29f9a19ac9f14577a5de672c45a0000e
I20260924 21:10:21.648623 15220 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260924 21:10:21.648741 15220 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fc2437869ad google::LogMessage::Fail() at ??:0
@ 0x7fc24378988a google::LogMessage::SendToLog() at ??:0
@ 0x7fc2437864de google::LogMessage::Flush() at ??:0
@ 0x7fc243787a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fc24b6ab0f1 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7fc24b6b48b9 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fc2456540d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fc245649954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fc2464ad6db start_thread at ??:0
@ 0x7fc24228461f clone at ??:0
W20260924 21:10:21.731917 15209 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 21:10:21.732074 15209 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 21:10:21.732092 15209 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 21:10:21.733105 15209 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 21:10:21.733144 15209 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 21:10:21.733167 15209 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 21:10:21.734535 15209 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35231
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790284076800039-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790284076800039-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790284076800039-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:44771
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790284076800039-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=38341
--tserver_master_addrs=127.0.98.62:45663
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15209
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790284076800039-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 19:49:06 UTC on ccycloud.quasar-vhqjoc.root.comops.site
build id 4670864
I20260924 21:10:21.734738 15209 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 21:10:21.734936 15209 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 21:10:21.739933 15243 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 21:10:21.739943 15244 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 21:10:21.739944 15246 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 21:10:21.740129 15209 server_base.cc:1034] running on GCE node
I20260924 21:10:21.740273 15209 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 21:10:21.740463 15209 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 21:10:21.741649 15209 hybrid_clock.cc:648] HybridClock initialized: now 1790284221741617 us; error 49 us; skew 500 ppm
Sep 24 21:10:21 dist-test-slave-c7tk krb5kdc[14071](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284221, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 21:10:21.743767 15209 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 21:10:21.744257 15209 webserver.cc:460] Webserver started at http://127.0.98.1:38341/ using document root <none> and password file <none>
I20260924 21:10:21.744478 15209 fs_manager.cc:359] Metadata directory not provided
I20260924 21:10:21.744537 15209 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 21:10:21.746031 15209 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260924 21:10:21.746673 15253 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 21:10:21.746919 15209 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260924 21:10:21.746994 15209 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790284076800039-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790284076800039-392-0/minicluster-data/ts-0/wal
uuid: "98dbb90780044a349f025e385e51e5c7"
format_stamp: "Formatted at 2026-09-24 21:10:20 on dist-test-slave-c7tk"
server_key: "ad8242e15fbb6b838111c50417b89741"
server_key_iv: "a745b8c1045ad53db9fd3d021a833161"
server_key_version: "encryptionkey@0"
I20260924 21:10:21.747365 15209 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790284076800039-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790284076800039-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790284076800039-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 21:10:21.760329 15209 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 21:10:21.762240 15209 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 21:10:21.762367 15209 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 21:10:21.762563 15209 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 21:10:21.762964 15260 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260924 21:10:21.764091 15209 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260924 21:10:21.764148 15209 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.001s user 0.000s sys 0.000s
I20260924 21:10:21.764180 15209 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260924 21:10:21.764933 15209 ts_tablet_manager.cc:610] Registered 1 tablets
I20260924 21:10:21.764974 15209 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260924 21:10:21.765033 15260 tablet_bootstrap.cc:492] T 44208eb9cbaa49269197c60f035e4d02 P 98dbb90780044a349f025e385e51e5c7: Bootstrap starting.
I20260924 21:10:21.771409 15209 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:44771
I20260924 21:10:21.771571 15367 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:44771 every 8 connection(s)
I20260924 21:10:21.771816 15209 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790284076800039-392-0/minicluster-data/ts-0/data/info.pb
I20260924 21:10:21.774559 15260 log.cc:824] T 44208eb9cbaa49269197c60f035e4d02 P 98dbb90780044a349f025e385e51e5c7: Log is configured to *not* fsync() on all Append() calls
I20260924 21:10:21.776237 15260 tablet_bootstrap.cc:492] T 44208eb9cbaa49269197c60f035e4d02 P 98dbb90780044a349f025e385e51e5c7: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 21:10:21.776475 15260 tablet_bootstrap.cc:492] T 44208eb9cbaa49269197c60f035e4d02 P 98dbb90780044a349f025e385e51e5c7: Bootstrap complete.
I20260924 21:10:21.776799 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAzNNjS/build/release/bin/kudu as pid 15209
I20260924 21:10:21.777076 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAzNNjS/build/release/bin/kudu
/tmp/dist-test-taskAzNNjS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790284076800039-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790284076800039-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790284076800039-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790284076800039-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:41909
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:45663
--webserver_port=46117
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790284076800039-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35231
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790284076800039-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790284076800039-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790284076800039-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 21:10:21.777395 15260 ts_tablet_manager.cc:1397] T 44208eb9cbaa49269197c60f035e4d02 P 98dbb90780044a349f025e385e51e5c7: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.004s
I20260924 21:10:21.778358 15260 raft_consensus.cc:348] T 44208eb9cbaa49269197c60f035e4d02 P 98dbb90780044a349f025e385e51e5c7 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "985271ad73854c8eb773b3da78a409dc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46821 } } peers { permanent_uuid: "6125c714aa5640fba1a78fb9c555e3d2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41909 } } peers { permanent_uuid: "98dbb90780044a349f025e385e51e5c7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44771 } }
I20260924 21:10:21.778690 15260 raft_consensus.cc:729] T 44208eb9cbaa49269197c60f035e4d02 P 98dbb90780044a349f025e385e51e5c7 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98dbb90780044a349f025e385e51e5c7, State: Initialized, Role: FOLLOWER
I20260924 21:10:21.778822 15260 consensus_queue.cc:260] T 44208eb9cbaa49269197c60f035e4d02 P 98dbb90780044a349f025e385e51e5c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "985271ad73854c8eb773b3da78a409dc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46821 } } peers { permanent_uuid: "6125c714aa5640fba1a78fb9c555e3d2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41909 } } peers { permanent_uuid: "98dbb90780044a349f025e385e51e5c7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44771 } }
I20260924 21:10:21.779052 15260 ts_tablet_manager.cc:1428] T 44208eb9cbaa49269197c60f035e4d02 P 98dbb90780044a349f025e385e51e5c7: Time spent starting tablet: real 0.002s user 0.004s sys 0.000s
W20260924 21:10:21.839893 15370 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 21:10:21.772390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45663 (local address 127.0.98.1:51789)
0924 21:10:21.772541 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0924 21:10:21.772576 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0924 21:10:21.773178 (+ 602us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 21:10:21.839705 (+ 66527us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45663: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":86,"threads_started":1}
W20260924 21:10:21.840127 15368 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:45663 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:45663: Client connection negotiation failed: client connection to 127.0.98.62:45663: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260924 21:10:21.840469 15370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:10:21.840327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45663 (local address 127.0.98.1:41167)
0924 21:10:21.840380 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0924 21:10:21.840440 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260924 21:10:21.840883 15370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:10:21.840777 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45663 (local address 127.0.98.1:60775)
0924 21:10:21.840835 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0924 21:10:21.840866 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260924 21:10:21.841001 15368 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260924 21:10:21.847329 15372 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 21:10:21.847512 15372 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 21:10:21.847553 15372 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 21:10:21.848556 15372 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 21:10:21.848613 15372 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 21:10:21.848652 15372 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 21:10:21.850033 15372 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35231
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790284076800039-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790284076800039-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790284076800039-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:41909
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790284076800039-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=46117
--tserver_master_addrs=127.0.98.62:45663
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15372
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790284076800039-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 19:49:06 UTC on ccycloud.quasar-vhqjoc.root.comops.site
build id 4670864
I20260924 21:10:21.850244 15372 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 21:10:21.850476 15372 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 21:10:21.855163 15381 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 21:10:21.855226 15378 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 21:10:21.855358 15372 server_base.cc:1034] running on GCE node
W20260924 21:10:21.855409 15379 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 21:10:21.855583 15372 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 21:10:21.855806 15372 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 21:10:21.856954 15372 hybrid_clock.cc:648] HybridClock initialized: now 1790284221856931 us; error 35 us; skew 500 ppm
Sep 24 21:10:21 dist-test-slave-c7tk krb5kdc[14071](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284221, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 21:10:21.859143 15372 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 21:10:21.859498 15372 webserver.cc:460] Webserver started at http://127.0.98.2:46117/ using document root <none> and password file <none>
I20260924 21:10:21.859688 15372 fs_manager.cc:359] Metadata directory not provided
I20260924 21:10:21.859746 15372 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 21:10:21.860870 15372 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 21:10:21.861485 15388 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 21:10:21.861644 15372 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 21:10:21.861726 15372 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790284076800039-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790284076800039-392-0/minicluster-data/ts-1/wal
uuid: "6125c714aa5640fba1a78fb9c555e3d2"
format_stamp: "Formatted at 2026-09-24 21:10:20 on dist-test-slave-c7tk"
server_key: "285202fe52dbb23bcbcc83d50a1c0497"
server_key_iv: "dfc07fb97f92866d307feec1d056a08e"
server_key_version: "encryptionkey@0"
I20260924 21:10:21.862031 15372 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790284076800039-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790284076800039-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790284076800039-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 21:10:21.873328 15372 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 21:10:21.874817 15372 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 21:10:21.874950 15372 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 21:10:21.875130 15372 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 21:10:21.875466 15395 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
W20260924 21:10:21.876487 15395 ts_tablet_manager.cc:1852] T 44208eb9cbaa49269197c60f035e4d02 P 6125c714aa5640fba1a78fb9c555e3d2: Tablet Manager startup: Rolling forward tablet deletion of type TABLET_DATA_DELETED
I20260924 21:10:21.876677 15395 ts_tablet_manager.cc:1905] T 44208eb9cbaa49269197c60f035e4d02 P 6125c714aa5640fba1a78fb9c555e3d2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 21:10:21.878721 15395 ts_tablet_manager.cc:1918] T 44208eb9cbaa49269197c60f035e4d02 P 6125c714aa5640fba1a78fb9c555e3d2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 21:10:21.878780 15395 log.cc:1196] T 44208eb9cbaa49269197c60f035e4d02 P 6125c714aa5640fba1a78fb9c555e3d2: Deleting WAL directory at /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790284076800039-392-0/minicluster-data/ts-1/wal/wals/44208eb9cbaa49269197c60f035e4d02
I20260924 21:10:21.878949 15395 ts_tablet_manager.cc:1939] T 44208eb9cbaa49269197c60f035e4d02 P 6125c714aa5640fba1a78fb9c555e3d2: Deleting consensus metadata
I20260924 21:10:21.879067 15372 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 0 live tablets)
I20260924 21:10:21.879122 15372 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.004s user 0.000s sys 0.000s
I20260924 21:10:21.879158 15372 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 21:10:21.879179 15372 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 21:10:21.884204 15372 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:41909
I20260924 21:10:21.884295 15502 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:41909 every 8 connection(s)
I20260924 21:10:21.884519 15372 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790284076800039-392-0/minicluster-data/ts-1/data/info.pb
I20260924 21:10:21.885286 15505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:10:21.885008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45663 (local address 127.0.98.2:58509)
0924 21:10:21.885138 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0924 21:10:21.885189 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":65,"threads_started":1}
W20260924 21:10:21.885473 15503 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:45663 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:45663: Client connection negotiation failed: client connection to 127.0.98.62:45663: connect: Connection refused (error 111)
I20260924 21:10:21.885700 15505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:10:21.885625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45663 (local address 127.0.98.2:53269)
0924 21:10:21.885667 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0924 21:10:21.885688 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260924 21:10:21.885958 15505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:10:21.885907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45663 (local address 127.0.98.2:59893)
0924 21:10:21.885932 (+ 25us) negotiation.cc:107] Waiting for socket to connect
0924 21:10:21.885948 (+ 16us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":12}
W20260924 21:10:21.886072 15503 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260924 21:10:21.891695 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAzNNjS/build/release/bin/kudu as pid 15372
I20260924 21:10:21.891904 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAzNNjS/build/release/bin/kudu
/tmp/dist-test-taskAzNNjS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790284076800039-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790284076800039-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790284076800039-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790284076800039-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:46821
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:45663
--webserver_port=40127
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790284076800039-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35231
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790284076800039-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790284076800039-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790284076800039-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 21:10:21.960065 15506 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 21:10:21.960212 15506 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 21:10:21.960232 15506 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 21:10:21.961331 15506 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 21:10:21.961378 15506 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 21:10:21.961429 15506 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 21:10:21.962725 15506 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35231
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790284076800039-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790284076800039-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790284076800039-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:46821
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790284076800039-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=40127
--tserver_master_addrs=127.0.98.62:45663
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15506
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790284076800039-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 19:49:06 UTC on ccycloud.quasar-vhqjoc.root.comops.site
build id 4670864
I20260924 21:10:21.962922 15506 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 21:10:21.963106 15506 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 21:10:21.967490 15511 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 21:10:21.967530 15512 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 21:10:21.967547 15514 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 21:10:21.967741 15506 server_base.cc:1034] running on GCE node
I20260924 21:10:21.967934 15506 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 21:10:21.968132 15506 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 21:10:21.969280 15506 hybrid_clock.cc:648] HybridClock initialized: now 1790284221969274 us; error 33 us; skew 500 ppm
Sep 24 21:10:21 dist-test-slave-c7tk krb5kdc[14071](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284221, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 21:10:21.971472 15506 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 21:10:21.971863 15506 webserver.cc:460] Webserver started at http://127.0.98.3:40127/ using document root <none> and password file <none>
I20260924 21:10:21.972060 15506 fs_manager.cc:359] Metadata directory not provided
I20260924 21:10:21.972146 15506 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 21:10:21.973276 15506 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 21:10:21.973847 15521 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 21:10:21.974030 15506 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 21:10:21.974118 15506 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790284076800039-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790284076800039-392-0/minicluster-data/ts-2/wal
uuid: "985271ad73854c8eb773b3da78a409dc"
format_stamp: "Formatted at 2026-09-24 21:10:20 on dist-test-slave-c7tk"
server_key: "08f89c81555537cd2f66dbbeaf5bf73e"
server_key_iv: "e72cffdb59c19b9a104557b64137e840"
server_key_version: "encryptionkey@0"
I20260924 21:10:21.974416 15506 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790284076800039-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790284076800039-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790284076800039-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 21:10:21.980338 15506 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 21:10:21.981973 15506 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 21:10:21.982096 15506 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 21:10:21.982264 15506 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 21:10:21.982507 15506 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 21:10:21.982558 15506 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 21:10:21.982589 15506 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 21:10:21.982611 15506 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 21:10:21.987701 15506 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:46821
I20260924 21:10:21.987730 15634 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:46821 every 8 connection(s)
I20260924 21:10:21.988004 15506 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790284076800039-392-0/minicluster-data/ts-2/data/info.pb
I20260924 21:10:21.988801 15637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:10:21.988520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45663 (local address 127.0.98.3:33109)
0924 21:10:21.988644 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0924 21:10:21.988696 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":59,"threads_started":1}
W20260924 21:10:21.989133 15635 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:45663 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:45663: Client connection negotiation failed: client connection to 127.0.98.62:45663: connect: Connection refused (error 111)
I20260924 21:10:21.989346 15637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:10:21.989269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45663 (local address 127.0.98.3:36337)
0924 21:10:21.989293 (+ 24us) negotiation.cc:107] Waiting for socket to connect
0924 21:10:21.989313 (+ 20us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":12}
I20260924 21:10:21.989585 15637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:10:21.989518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45663 (local address 127.0.98.3:41561)
0924 21:10:21.989557 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0924 21:10:21.989574 (+ 17us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260924 21:10:21.989684 15635 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260924 21:10:21.995862 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAzNNjS/build/release/bin/kudu as pid 15506
I20260924 21:10:22.841553 15639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:10:22.841313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45663 (local address 127.0.98.1:41681)
0924 21:10:22.841458 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0924 21:10:22.841492 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":69,"threads_started":1}
I20260924 21:10:22.886485 15640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:10:22.886323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45663 (local address 127.0.98.2:53895)
0924 21:10:22.886427 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0924 21:10:22.886456 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91,"thread_start_us":53,"threads_started":1}
I20260924 21:10:22.990211 15641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:10:22.990024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45663 (local address 127.0.98.3:41179)
0924 21:10:22.990135 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0924 21:10:22.990166 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":56,"threads_started":1}
I20260924 21:10:22.998813 15642 raft_consensus.cc:482] T 44208eb9cbaa49269197c60f035e4d02 P 98dbb90780044a349f025e385e51e5c7 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 21:10:22.998903 15642 raft_consensus.cc:504] T 44208eb9cbaa49269197c60f035e4d02 P 98dbb90780044a349f025e385e51e5c7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "985271ad73854c8eb773b3da78a409dc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46821 } } peers { permanent_uuid: "6125c714aa5640fba1a78fb9c555e3d2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41909 } } peers { permanent_uuid: "98dbb90780044a349f025e385e51e5c7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44771 } }
I20260924 21:10:22.999254 15642 leader_election.cc:288] T 44208eb9cbaa49269197c60f035e4d02 P 98dbb90780044a349f025e385e51e5c7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 985271ad73854c8eb773b3da78a409dc (127.0.98.3:46821), 6125c714aa5640fba1a78fb9c555e3d2 (127.0.98.2:41909)
Sep 24 21:10:23 dist-test-slave-c7tk krb5kdc[14071](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284221, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.2@KRBTEST.COM
Sep 24 21:10:23 dist-test-slave-c7tk krb5kdc[14071](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284221, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.3@KRBTEST.COM
I20260924 21:10:23.008667 15639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:10:22.999372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:46821 (local address 127.0.98.1:54253)
0924 21:10:22.999441 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0924 21:10:22.999452 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 21:10:22.999506 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 21:10:23.000228 (+ 722us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 21:10:23.000231 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 21:10:23.000602 (+ 371us) client_negotiation.cc:189] Negotiated authn=SASL
0924 21:10:23.000790 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:10:23.000795 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:10:23.001856 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:10:23.001857 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:10:23.002614 (+ 757us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:10:23.002616 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:10:23.003213 (+ 597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:10:23.003214 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:10:23.003327 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:10:23.004004 (+ 677us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 21:10:23.004008 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 21:10:23.005762 (+ 1754us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 21:10:23.007389 (+ 1627us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 21:10:23.007390 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 21:10:23.007390 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 21:10:23.007576 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 21:10:23.007799 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 21:10:23.007800 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 21:10:23.007800 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 21:10:23.007908 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 21:10:23.008231 (+ 323us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 21:10:23.008236 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 21:10:23.008347 (+ 111us) client_negotiation.cc:769] Sending connection context
0924 21:10:23.008432 (+ 85us) client_negotiation.cc:240] Negotiation successful
0924 21:10:23.008585 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52,"mutex_wait_us":14}
I20260924 21:10:23.008677 15645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:10:22.999529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:42529 (local address 127.0.98.2:41909)
0924 21:10:22.999667 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0924 21:10:22.999676 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 21:10:22.999699 (+ 23us) server_negotiation.cc:407] Connection header received
0924 21:10:23.000116 (+ 417us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:10:23.000118 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:10:23.000152 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:10:23.000293 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:10:23.000909 (+ 616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:10:23.001752 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:10:23.002697 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:10:23.003100 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:10:23.003236 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:10:23.005490 (+ 2254us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 21:10:23.005493 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 21:10:23.005493 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 21:10:23.005502 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 21:10:23.007247 (+ 1745us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:10:23.007650 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:10:23.007653 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:10:23.007653 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:10:23.007732 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:10:23.007967 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:10:23.007968 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:10:23.007969 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:10:23.008159 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 21:10:23.008236 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0924 21:10:23.008483 (+ 247us) server_negotiation.cc:299] Negotiation successful
0924 21:10:23.008585 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108,"thread_start_us":87,"threads_started":1}
I20260924 21:10:23.008708 15644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:10:22.999470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:54253 (local address 127.0.98.3:46821)
0924 21:10:22.999588 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0924 21:10:22.999600 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 21:10:22.999628 (+ 28us) server_negotiation.cc:407] Connection header received
0924 21:10:23.000015 (+ 387us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:10:23.000017 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:10:23.000052 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:10:23.000197 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:10:23.000909 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:10:23.001733 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:10:23.002718 (+ 985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:10:23.003102 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:10:23.003241 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:10:23.005880 (+ 2639us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 21:10:23.005883 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 21:10:23.005883 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 21:10:23.005892 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 21:10:23.007300 (+ 1408us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:10:23.007650 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:10:23.007652 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:10:23.007653 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:10:23.007731 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:10:23.007965 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:10:23.007966 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:10:23.007966 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:10:23.008159 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 21:10:23.008236 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0924 21:10:23.008498 (+ 262us) server_negotiation.cc:299] Negotiation successful
0924 21:10:23.008617 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92,"thread_start_us":40,"threads_started":1}
I20260924 21:10:23.008767 15643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:10:22.999394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41909 (local address 127.0.98.1:42529)
0924 21:10:22.999509 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0924 21:10:22.999519 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 21:10:22.999569 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 21:10:23.000272 (+ 703us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 21:10:23.000272 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 21:10:23.000601 (+ 329us) client_negotiation.cc:189] Negotiated authn=SASL
0924 21:10:23.000790 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:10:23.000795 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:10:23.001924 (+ 1129us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:10:23.001925 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:10:23.002616 (+ 691us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:10:23.002618 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:10:23.003214 (+ 596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:10:23.003215 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:10:23.003316 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:10:23.003948 (+ 632us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 21:10:23.003956 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 21:10:23.005401 (+ 1445us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 21:10:23.007320 (+ 1919us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 21:10:23.007326 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 21:10:23.007328 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 21:10:23.007577 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 21:10:23.007802 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 21:10:23.007803 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 21:10:23.007803 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 21:10:23.007910 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 21:10:23.008234 (+ 324us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 21:10:23.008239 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 21:10:23.008346 (+ 107us) client_negotiation.cc:769] Sending connection context
0924 21:10:23.008432 (+ 86us) client_negotiation.cc:240] Negotiation successful
0924 21:10:23.008654 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103,"mutex_wait_us":159,"thread_start_us":55,"threads_started":1}
W20260924 21:10:23.009859 15256 leader_election.cc:341] T 44208eb9cbaa49269197c60f035e4d02 P 98dbb90780044a349f025e385e51e5c7 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6125c714aa5640fba1a78fb9c555e3d2 (127.0.98.2:41909): Not found: Tablet not found: 44208eb9cbaa49269197c60f035e4d02
W20260924 21:10:23.009922 15255 leader_election.cc:341] T 44208eb9cbaa49269197c60f035e4d02 P 98dbb90780044a349f025e385e51e5c7 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 985271ad73854c8eb773b3da78a409dc (127.0.98.3:46821): Not found: Tablet not found: 44208eb9cbaa49269197c60f035e4d02
I20260924 21:10:23.009945 15256 leader_election.cc:302] T 44208eb9cbaa49269197c60f035e4d02 P 98dbb90780044a349f025e385e51e5c7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 98dbb90780044a349f025e385e51e5c7; no voters: 6125c714aa5640fba1a78fb9c555e3d2, 985271ad73854c8eb773b3da78a409dc
I20260924 21:10:23.010032 15642 raft_consensus.cc:2726] T 44208eb9cbaa49269197c60f035e4d02 P 98dbb90780044a349f025e385e51e5c7 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 21:10:23.842176 15646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:10:23.842001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45663 (local address 127.0.98.1:42389)
0924 21:10:23.842108 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0924 21:10:23.842138 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":56,"threads_started":1}
I20260924 21:10:23.886986 15647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:10:23.886841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45663 (local address 127.0.98.2:34877)
0924 21:10:23.886931 (+ 90us) negotiation.cc:107] Waiting for socket to connect
0924 21:10:23.886956 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78,"thread_start_us":45,"threads_started":1}
I20260924 21:10:23.990722 15648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:10:23.990562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45663 (local address 127.0.98.3:49733)
0924 21:10:23.990662 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0924 21:10:23.990693 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87,"thread_start_us":53,"threads_started":1}
I20260924 21:10:24.842808 15649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:10:24.842626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45663 (local address 127.0.98.1:60419)
0924 21:10:24.842737 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0924 21:10:24.842769 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":53,"threads_started":1}
I20260924 21:10:24.887516 15650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:10:24.887344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45663 (local address 127.0.98.2:40377)
0924 21:10:24.887454 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0924 21:10:24.887483 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":57,"threads_started":1}
I20260924 21:10:24.991308 15651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:10:24.991105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45663 (local address 127.0.98.3:56979)
0924 21:10:24.991217 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0924 21:10:24.991252 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":54,"threads_started":1}
I20260924 21:10:25.183992 15652 raft_consensus.cc:482] T 44208eb9cbaa49269197c60f035e4d02 P 98dbb90780044a349f025e385e51e5c7 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 21:10:25.184065 15652 raft_consensus.cc:504] T 44208eb9cbaa49269197c60f035e4d02 P 98dbb90780044a349f025e385e51e5c7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "985271ad73854c8eb773b3da78a409dc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46821 } } peers { permanent_uuid: "6125c714aa5640fba1a78fb9c555e3d2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41909 } } peers { permanent_uuid: "98dbb90780044a349f025e385e51e5c7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44771 } }
I20260924 21:10:25.184288 15652 leader_election.cc:288] T 44208eb9cbaa49269197c60f035e4d02 P 98dbb90780044a349f025e385e51e5c7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 985271ad73854c8eb773b3da78a409dc (127.0.98.3:46821), 6125c714aa5640fba1a78fb9c555e3d2 (127.0.98.2:41909)
W20260924 21:10:25.184643 15255 leader_election.cc:341] T 44208eb9cbaa49269197c60f035e4d02 P 98dbb90780044a349f025e385e51e5c7 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 985271ad73854c8eb773b3da78a409dc (127.0.98.3:46821): Not found: Tablet not found: 44208eb9cbaa49269197c60f035e4d02
W20260924 21:10:25.184743 15256 leader_election.cc:341] T 44208eb9cbaa49269197c60f035e4d02 P 98dbb90780044a349f025e385e51e5c7 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6125c714aa5640fba1a78fb9c555e3d2 (127.0.98.2:41909): Not found: Tablet not found: 44208eb9cbaa49269197c60f035e4d02
I20260924 21:10:25.184809 15256 leader_election.cc:302] T 44208eb9cbaa49269197c60f035e4d02 P 98dbb90780044a349f025e385e51e5c7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 98dbb90780044a349f025e385e51e5c7; no voters: 6125c714aa5640fba1a78fb9c555e3d2, 985271ad73854c8eb773b3da78a409dc
I20260924 21:10:25.184896 15652 raft_consensus.cc:2726] T 44208eb9cbaa49269197c60f035e4d02 P 98dbb90780044a349f025e385e51e5c7 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 21:10:25.843411 15653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:10:25.843215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45663 (local address 127.0.98.1:46663)
0924 21:10:25.843333 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0924 21:10:25.843365 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":51,"threads_started":1}
I20260924 21:10:25.888115 15654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:10:25.887908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45663 (local address 127.0.98.2:51797)
0924 21:10:25.888045 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0924 21:10:25.888079 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":67,"threads_started":1}
I20260924 21:10:25.991879 15655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:10:25.991695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45663 (local address 127.0.98.3:48077)
0924 21:10:25.991807 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0924 21:10:25.991838 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":58,"threads_started":1}
I20260924 21:10:26.844094 15656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:10:26.843907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45663 (local address 127.0.98.1:52419)
0924 21:10:26.844017 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0924 21:10:26.844050 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":53,"threads_started":1}
I20260924 21:10:26.888660 15657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:10:26.888517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45663 (local address 127.0.98.2:47627)
0924 21:10:26.888607 (+ 90us) negotiation.cc:107] Waiting for socket to connect
0924 21:10:26.888633 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79,"thread_start_us":45,"threads_started":1}
I20260924 21:10:26.992488 15658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:10:26.992323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45663 (local address 127.0.98.3:59179)
0924 21:10:26.992434 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0924 21:10:26.992459 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":63,"threads_started":1}
I20260924 21:10:27.608937 15659 raft_consensus.cc:482] T 44208eb9cbaa49269197c60f035e4d02 P 98dbb90780044a349f025e385e51e5c7 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 21:10:27.609020 15659 raft_consensus.cc:504] T 44208eb9cbaa49269197c60f035e4d02 P 98dbb90780044a349f025e385e51e5c7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "985271ad73854c8eb773b3da78a409dc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46821 } } peers { permanent_uuid: "6125c714aa5640fba1a78fb9c555e3d2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41909 } } peers { permanent_uuid: "98dbb90780044a349f025e385e51e5c7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44771 } }
I20260924 21:10:27.609237 15659 leader_election.cc:288] T 44208eb9cbaa49269197c60f035e4d02 P 98dbb90780044a349f025e385e51e5c7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 985271ad73854c8eb773b3da78a409dc (127.0.98.3:46821), 6125c714aa5640fba1a78fb9c555e3d2 (127.0.98.2:41909)
W20260924 21:10:27.609643 15256 leader_election.cc:341] T 44208eb9cbaa49269197c60f035e4d02 P 98dbb90780044a349f025e385e51e5c7 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6125c714aa5640fba1a78fb9c555e3d2 (127.0.98.2:41909): Not found: Tablet not found: 44208eb9cbaa49269197c60f035e4d02
W20260924 21:10:27.609757 15255 leader_election.cc:341] T 44208eb9cbaa49269197c60f035e4d02 P 98dbb90780044a349f025e385e51e5c7 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 985271ad73854c8eb773b3da78a409dc (127.0.98.3:46821): Not found: Tablet not found: 44208eb9cbaa49269197c60f035e4d02
I20260924 21:10:27.609814 15255 leader_election.cc:302] T 44208eb9cbaa49269197c60f035e4d02 P 98dbb90780044a349f025e385e51e5c7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 98dbb90780044a349f025e385e51e5c7; no voters: 6125c714aa5640fba1a78fb9c555e3d2, 985271ad73854c8eb773b3da78a409dc
I20260924 21:10:27.609908 15659 raft_consensus.cc:2726] T 44208eb9cbaa49269197c60f035e4d02 P 98dbb90780044a349f025e385e51e5c7 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 21:10:27.844792 15660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:10:27.844600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45663 (local address 127.0.98.1:54907)
0924 21:10:27.844717 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0924 21:10:27.844746 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":77,"threads_started":1}
I20260924 21:10:27.889178 15661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:10:27.889001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45663 (local address 127.0.98.2:34323)
0924 21:10:27.889111 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0924 21:10:27.889143 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":50,"threads_started":1}
I20260924 21:10:27.993069 15662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:10:27.992889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45663 (local address 127.0.98.3:57935)
0924 21:10:27.993000 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0924 21:10:27.993037 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":55,"threads_started":1}
I20260924 21:10:28.845394 15663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:10:28.845201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45663 (local address 127.0.98.1:55693)
0924 21:10:28.845313 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0924 21:10:28.845352 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":51,"threads_started":1}
I20260924 21:10:28.889758 15664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:10:28.889578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45663 (local address 127.0.98.2:49823)
0924 21:10:28.889695 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0924 21:10:28.889727 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":53,"threads_started":1}
I20260924 21:10:28.993672 15665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:10:28.993486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45663 (local address 127.0.98.3:42569)
0924 21:10:28.993609 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0924 21:10:28.993639 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":71,"threads_started":1}
I20260924 21:10:29.245033 15666 raft_consensus.cc:482] T 44208eb9cbaa49269197c60f035e4d02 P 98dbb90780044a349f025e385e51e5c7 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 21:10:29.245182 15666 raft_consensus.cc:504] T 44208eb9cbaa49269197c60f035e4d02 P 98dbb90780044a349f025e385e51e5c7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "985271ad73854c8eb773b3da78a409dc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46821 } } peers { permanent_uuid: "6125c714aa5640fba1a78fb9c555e3d2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41909 } } peers { permanent_uuid: "98dbb90780044a349f025e385e51e5c7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44771 } }
I20260924 21:10:29.245364 15666 leader_election.cc:288] T 44208eb9cbaa49269197c60f035e4d02 P 98dbb90780044a349f025e385e51e5c7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 985271ad73854c8eb773b3da78a409dc (127.0.98.3:46821), 6125c714aa5640fba1a78fb9c555e3d2 (127.0.98.2:41909)
W20260924 21:10:29.245702 15256 leader_election.cc:341] T 44208eb9cbaa49269197c60f035e4d02 P 98dbb90780044a349f025e385e51e5c7 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6125c714aa5640fba1a78fb9c555e3d2 (127.0.98.2:41909): Not found: Tablet not found: 44208eb9cbaa49269197c60f035e4d02
W20260924 21:10:29.245777 15255 leader_election.cc:341] T 44208eb9cbaa49269197c60f035e4d02 P 98dbb90780044a349f025e385e51e5c7 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 985271ad73854c8eb773b3da78a409dc (127.0.98.3:46821): Not found: Tablet not found: 44208eb9cbaa49269197c60f035e4d02
I20260924 21:10:29.245862 15255 leader_election.cc:302] T 44208eb9cbaa49269197c60f035e4d02 P 98dbb90780044a349f025e385e51e5c7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 98dbb90780044a349f025e385e51e5c7; no voters: 6125c714aa5640fba1a78fb9c555e3d2, 985271ad73854c8eb773b3da78a409dc
I20260924 21:10:29.245954 15666 raft_consensus.cc:2726] T 44208eb9cbaa49269197c60f035e4d02 P 98dbb90780044a349f025e385e51e5c7 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 21:10:29.846089 15667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:10:29.845898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45663 (local address 127.0.98.1:36731)
0924 21:10:29.845997 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0924 21:10:29.846039 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86,"thread_start_us":48,"threads_started":1}
I20260924 21:10:29.890430 15668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:10:29.890222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45663 (local address 127.0.98.2:34029)
0924 21:10:29.890361 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0924 21:10:29.890400 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":66,"threads_started":1}
I20260924 21:10:29.994210 15669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:10:29.994042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45663 (local address 127.0.98.3:36117)
0924 21:10:29.994150 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0924 21:10:29.994178 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":56,"threads_started":1}
I20260924 21:10:30.846664 15670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:10:30.846471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45663 (local address 127.0.98.1:59837)
0924 21:10:30.846599 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0924 21:10:30.846624 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":80,"threads_started":1}
I20260924 21:10:30.891086 15671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:10:30.890897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45663 (local address 127.0.98.2:59067)
0924 21:10:30.891025 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0924 21:10:30.891057 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":60,"threads_started":1}
I20260924 21:10:30.994817 15672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:10:30.994608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45663 (local address 127.0.98.3:36349)
0924 21:10:30.994751 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0924 21:10:30.994783 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":74,"threads_started":1}
I20260924 21:10:31.556494 15673 raft_consensus.cc:482] T 44208eb9cbaa49269197c60f035e4d02 P 98dbb90780044a349f025e385e51e5c7 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 21:10:31.556594 15673 raft_consensus.cc:504] T 44208eb9cbaa49269197c60f035e4d02 P 98dbb90780044a349f025e385e51e5c7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "985271ad73854c8eb773b3da78a409dc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46821 } } peers { permanent_uuid: "6125c714aa5640fba1a78fb9c555e3d2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41909 } } peers { permanent_uuid: "98dbb90780044a349f025e385e51e5c7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44771 } }
I20260924 21:10:31.556769 15673 leader_election.cc:288] T 44208eb9cbaa49269197c60f035e4d02 P 98dbb90780044a349f025e385e51e5c7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 985271ad73854c8eb773b3da78a409dc (127.0.98.3:46821), 6125c714aa5640fba1a78fb9c555e3d2 (127.0.98.2:41909)
W20260924 21:10:31.557122 15255 leader_election.cc:341] T 44208eb9cbaa49269197c60f035e4d02 P 98dbb90780044a349f025e385e51e5c7 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 985271ad73854c8eb773b3da78a409dc (127.0.98.3:46821): Not found: Tablet not found: 44208eb9cbaa49269197c60f035e4d02
W20260924 21:10:31.557186 15256 leader_election.cc:341] T 44208eb9cbaa49269197c60f035e4d02 P 98dbb90780044a349f025e385e51e5c7 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6125c714aa5640fba1a78fb9c555e3d2 (127.0.98.2:41909): Not found: Tablet not found: 44208eb9cbaa49269197c60f035e4d02
I20260924 21:10:31.557224 15256 leader_election.cc:302] T 44208eb9cbaa49269197c60f035e4d02 P 98dbb90780044a349f025e385e51e5c7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 98dbb90780044a349f025e385e51e5c7; no voters: 6125c714aa5640fba1a78fb9c555e3d2, 985271ad73854c8eb773b3da78a409dc
I20260924 21:10:31.557304 15673 raft_consensus.cc:2726] T 44208eb9cbaa49269197c60f035e4d02 P 98dbb90780044a349f025e385e51e5c7 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 21:10:31.847215 15674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:10:31.847027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45663 (local address 127.0.98.1:53177)
0924 21:10:31.847148 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0924 21:10:31.847177 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":65,"threads_started":1}
I20260924 21:10:31.891639 15675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:10:31.891460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45663 (local address 127.0.98.2:58701)
0924 21:10:31.891573 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0924 21:10:31.891606 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":59,"threads_started":1}
I20260924 21:10:31.995426 15676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:10:31.995248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45663 (local address 127.0.98.3:53555)
0924 21:10:31.995366 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0924 21:10:31.995394 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":70,"threads_started":1}
I20260924 21:10:32.847762 15677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:10:32.847583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45663 (local address 127.0.98.1:37137)
0924 21:10:32.847690 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0924 21:10:32.847723 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93,"thread_start_us":66,"threads_started":1}
I20260924 21:10:32.892253 15678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:10:32.892075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45663 (local address 127.0.98.2:36609)
0924 21:10:32.892194 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0924 21:10:32.892224 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":67,"threads_started":1}
I20260924 21:10:32.995937 15679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:10:32.995781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45663 (local address 127.0.98.3:48391)
0924 21:10:32.995878 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0924 21:10:32.995907 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84,"thread_start_us":47,"threads_started":1}
I20260924 21:10:33.848459 15680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:10:33.848262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45663 (local address 127.0.98.1:47075)
0924 21:10:33.848379 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0924 21:10:33.848418 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":53,"threads_started":1}
I20260924 21:10:33.892840 15681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:10:33.892640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45663 (local address 127.0.98.2:51781)
0924 21:10:33.892759 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0924 21:10:33.892806 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":50,"threads_started":1}
I20260924 21:10:33.996564 15682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:10:33.996389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45663 (local address 127.0.98.3:51479)
0924 21:10:33.996503 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0924 21:10:33.996533 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":60,"threads_started":1}
I20260924 21:10:34.849171 15683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:10:34.848883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45663 (local address 127.0.98.1:38689)
0924 21:10:34.849100 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0924 21:10:34.849132 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":61,"threads_started":1}
I20260924 21:10:34.893431 15684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:10:34.893236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45663 (local address 127.0.98.2:51875)
0924 21:10:34.893371 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0924 21:10:34.893402 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":64,"threads_started":1}
I20260924 21:10:34.997073 15685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:10:34.996925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45663 (local address 127.0.98.3:54301)
0924 21:10:34.997018 (+ 93us) negotiation.cc:107] Waiting for socket to connect
0924 21:10:34.997044 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80,"thread_start_us":48,"threads_started":1}
I20260924 21:10:35.849920 15686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:10:35.849700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45663 (local address 127.0.98.1:56235)
0924 21:10:35.849845 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0924 21:10:35.849877 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":68,"threads_started":1}
I20260924 21:10:35.894088 15687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:10:35.893908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45663 (local address 127.0.98.2:44943)
0924 21:10:35.894023 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0924 21:10:35.894057 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":56,"threads_started":1}
I20260924 21:10:35.997617 15688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:10:35.997444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45663 (local address 127.0.98.3:39215)
0924 21:10:35.997557 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0924 21:10:35.997587 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":63,"threads_started":1}
I20260924 21:10:36.850682 15689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:10:36.850435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45663 (local address 127.0.98.1:37437)
0924 21:10:36.850559 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0924 21:10:36.850626 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":75,"threads_started":1}
I20260924 21:10:36.894685 15690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:10:36.894497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45663 (local address 127.0.98.2:44629)
0924 21:10:36.894619 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0924 21:10:36.894649 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":65,"threads_started":1}
I20260924 21:10:36.997864 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAzNNjS/build/release/bin/kudu with pid 15209
I20260924 21:10:36.998656 15691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:10:36.998491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45663 (local address 127.0.98.3:43737)
0924 21:10:36.998593 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0924 21:10:36.998617 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86,"thread_start_us":47,"threads_started":1}
I20260924 21:10:37.002444 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAzNNjS/build/release/bin/kudu with pid 15372
I20260924 21:10:37.006263 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAzNNjS/build/release/bin/kudu with pid 15506
2026-09-24T21:10:37Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (17921 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790284076800039-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 21:10:37 dist-test-slave-c7tk krb5kdc[15694](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 21:10:37 dist-test-slave-c7tk krb5kdc[15694](info): set up 2 sockets
Sep 24 21:10:37 dist-test-slave-c7tk krb5kdc[15694](info): commencing operation
krb5kdc: starting...
W20260924 21:10:37.941275 392 mini_kdc.cc:121] Time spent starting KDC: real 0.916s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790284076800039-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790284076800039-392-0/krb5kdc/test-user.keytab.
Sep 24 21:10:38 dist-test-slave-c7tk krb5kdc[15694](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284238, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T21:10:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T21:10:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790284076800039-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790284076800039-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 21:10:38.040748 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAzNNjS/build/release/bin/kudu
/tmp/dist-test-taskAzNNjS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790284076800039-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790284076800039-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790284076800039-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790284076800039-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39885
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790284076800039-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44143
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39885
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790284076800039-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790284076800039-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790284076800039-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 21:10:38.113111 15707 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 21:10:38.113333 15707 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 21:10:38.113369 15707 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 21:10:38.114487 15707 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260924 21:10:38.114540 15707 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260924 21:10:38.114557 15707 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260924 21:10:38.114574 15707 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 21:10:38.114599 15707 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 21:10:38.114624 15707 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 21:10:38.114645 15707 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 21:10:38.114670 15707 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 21:10:38.115860 15707 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44143
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790284076800039-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790284076800039-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39885
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790284076800039-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39885
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790284076800039-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15707
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790284076800039-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 19:49:06 UTC on ccycloud.quasar-vhqjoc.root.comops.site
build id 4670864
I20260924 21:10:38.116083 15707 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 21:10:38.116262 15707 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 21:10:38.120278 15715 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 21:10:38.120292 15713 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 21:10:38.120376 15707 server_base.cc:1034] running on GCE node
W20260924 21:10:38.120292 15712 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 21:10:38.120771 15707 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 21:10:38.121011 15707 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 21:10:38.122159 15707 hybrid_clock.cc:648] HybridClock initialized: now 1790284238122144 us; error 40 us; skew 500 ppm
Sep 24 21:10:38 dist-test-slave-c7tk krb5kdc[15694](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284238, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 21:10:38.124341 15707 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 21:10:38.124743 15707 webserver.cc:460] Webserver started at http://127.0.98.62:38395/ using document root <none> and password file <none>
I20260924 21:10:38.124920 15707 fs_manager.cc:359] Metadata directory not provided
I20260924 21:10:38.124974 15707 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 21:10:38.125105 15707 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 21:10:38.126017 15707 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790284076800039-392-0/minicluster-data/master-0/data/instance:
uuid: "a18533bad310410ca2873fde2d8ff491"
format_stamp: "Formatted at 2026-09-24 21:10:38 on dist-test-slave-c7tk"
server_key: "7a801cb9420e22bcf66f976e7e25d2fd"
server_key_iv: "db9a9a6f8c42619d34d226e73332aed5"
server_key_version: "encryptionkey@0"
I20260924 21:10:38.126322 15707 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790284076800039-392-0/minicluster-data/master-0/wal/instance:
uuid: "a18533bad310410ca2873fde2d8ff491"
format_stamp: "Formatted at 2026-09-24 21:10:38 on dist-test-slave-c7tk"
server_key: "7a801cb9420e22bcf66f976e7e25d2fd"
server_key_iv: "db9a9a6f8c42619d34d226e73332aed5"
server_key_version: "encryptionkey@0"
I20260924 21:10:38.127377 15707 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 21:10:38.127971 15722 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 21:10:38.128114 15707 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 21:10:38.128196 15707 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790284076800039-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790284076800039-392-0/minicluster-data/master-0/wal
uuid: "a18533bad310410ca2873fde2d8ff491"
format_stamp: "Formatted at 2026-09-24 21:10:38 on dist-test-slave-c7tk"
server_key: "7a801cb9420e22bcf66f976e7e25d2fd"
server_key_iv: "db9a9a6f8c42619d34d226e73332aed5"
server_key_version: "encryptionkey@0"
I20260924 21:10:38.128293 15707 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790284076800039-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790284076800039-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790284076800039-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 21:10:38.134776 15707 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 21:10:38.136128 15707 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 21:10:38.136255 15707 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 21:10:38.139506 15707 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39885
I20260924 21:10:38.139540 15774 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39885 every 8 connection(s)
I20260924 21:10:38.139815 15707 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790284076800039-392-0/minicluster-data/master-0/data/info.pb
I20260924 21:10:38.140506 15775 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 21:10:38.142491 15775 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a18533bad310410ca2873fde2d8ff491: Bootstrap starting.
I20260924 21:10:38.143013 15775 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a18533bad310410ca2873fde2d8ff491: Neither blocks nor log segments found. Creating new log.
I20260924 21:10:38.143329 15775 log.cc:824] T 00000000000000000000000000000000 P a18533bad310410ca2873fde2d8ff491: Log is configured to *not* fsync() on all Append() calls
I20260924 21:10:38.143895 15775 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a18533bad310410ca2873fde2d8ff491: No bootstrap required, opened a new log
I20260924 21:10:38.144977 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAzNNjS/build/release/bin/kudu as pid 15707
I20260924 21:10:38.145072 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790284076800039-392-0/minicluster-data/master-0/wal/instance
I20260924 21:10:38.145233 392 external_mini_cluster.cc:1428] Setting key 50aa369368240896dc45bd44540ff8d7
I20260924 21:10:38.145259 15775 raft_consensus.cc:348] T 00000000000000000000000000000000 P a18533bad310410ca2873fde2d8ff491 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a18533bad310410ca2873fde2d8ff491" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39885 } }
I20260924 21:10:38.145437 15775 raft_consensus.cc:374] T 00000000000000000000000000000000 P a18533bad310410ca2873fde2d8ff491 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 21:10:38.145484 15775 raft_consensus.cc:729] T 00000000000000000000000000000000 P a18533bad310410ca2873fde2d8ff491 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a18533bad310410ca2873fde2d8ff491, State: Initialized, Role: FOLLOWER
I20260924 21:10:38.145612 15775 consensus_queue.cc:260] T 00000000000000000000000000000000 P a18533bad310410ca2873fde2d8ff491 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a18533bad310410ca2873fde2d8ff491" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39885 } }
I20260924 21:10:38.145689 15775 raft_consensus.cc:388] T 00000000000000000000000000000000 P a18533bad310410ca2873fde2d8ff491 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 21:10:38.145720 15775 raft_consensus.cc:482] T 00000000000000000000000000000000 P a18533bad310410ca2873fde2d8ff491 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 21:10:38.145761 15775 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a18533bad310410ca2873fde2d8ff491 [term 0 FOLLOWER]: Advancing to term 1
I20260924 21:10:38.146636 15775 raft_consensus.cc:504] T 00000000000000000000000000000000 P a18533bad310410ca2873fde2d8ff491 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a18533bad310410ca2873fde2d8ff491" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39885 } }
I20260924 21:10:38.146735 15775 leader_election.cc:302] T 00000000000000000000000000000000 P a18533bad310410ca2873fde2d8ff491 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a18533bad310410ca2873fde2d8ff491; no voters:
I20260924 21:10:38.146875 15775 leader_election.cc:288] T 00000000000000000000000000000000 P a18533bad310410ca2873fde2d8ff491 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 21:10:38.147038 15781 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a18533bad310410ca2873fde2d8ff491 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 21:10:38.147112 15775 sys_catalog.cc:564] T 00000000000000000000000000000000 P a18533bad310410ca2873fde2d8ff491 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 21:10:38.147145 15781 raft_consensus.cc:686] T 00000000000000000000000000000000 P a18533bad310410ca2873fde2d8ff491 [term 1 LEADER]: Becoming Leader. State: Replica: a18533bad310410ca2873fde2d8ff491, State: Running, Role: LEADER
I20260924 21:10:38.147212 15781 consensus_queue.cc:237] T 00000000000000000000000000000000 P a18533bad310410ca2873fde2d8ff491 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a18533bad310410ca2873fde2d8ff491" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39885 } }
I20260924 21:10:38.147487 15782 sys_catalog.cc:455] T 00000000000000000000000000000000 P a18533bad310410ca2873fde2d8ff491 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a18533bad310410ca2873fde2d8ff491" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a18533bad310410ca2873fde2d8ff491" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39885 } } }
I20260924 21:10:38.147497 15781 sys_catalog.cc:455] T 00000000000000000000000000000000 P a18533bad310410ca2873fde2d8ff491 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a18533bad310410ca2873fde2d8ff491. Latest consensus state: current_term: 1 leader_uuid: "a18533bad310410ca2873fde2d8ff491" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a18533bad310410ca2873fde2d8ff491" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39885 } } }
I20260924 21:10:38.147619 15782 sys_catalog.cc:458] T 00000000000000000000000000000000 P a18533bad310410ca2873fde2d8ff491 [sys.catalog]: This master's current role is: LEADER
I20260924 21:10:38.147636 15781 sys_catalog.cc:458] T 00000000000000000000000000000000 P a18533bad310410ca2873fde2d8ff491 [sys.catalog]: This master's current role is: LEADER
I20260924 21:10:38.147922 15789 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 21:10:38.148367 15789 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 21:10:38.150795 15789 catalog_manager.cc:1347] Generated new cluster ID: 00ea91f509a845098739aef3b1bf5fff
I20260924 21:10:38.150853 15789 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 24 21:10:38 dist-test-slave-c7tk krb5kdc[15694](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284238, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 21:10:38.155110 15779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:10:38.145758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46300 (local address 127.0.98.62:39885)
0924 21:10:38.145928 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0924 21:10:38.145934 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 21:10:38.145957 (+ 23us) server_negotiation.cc:407] Connection header received
0924 21:10:38.146032 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:10:38.146036 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:10:38.146067 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:10:38.146149 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:10:38.146922 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:10:38.148097 (+ 1175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:10:38.149102 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:10:38.149537 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:10:38.149639 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:10:38.151872 (+ 2233us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 21:10:38.151876 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 21:10:38.151876 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 21:10:38.151886 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0924 21:10:38.153485 (+ 1599us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:10:38.153847 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:10:38.153850 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:10:38.153850 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:10:38.154194 (+ 344us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:10:38.154405 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:10:38.154406 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:10:38.154406 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:10:38.154597 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 21:10:38.154721 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0924 21:10:38.154829 (+ 108us) server_negotiation.cc:299] Negotiation successful
0924 21:10:38.154947 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":100,"threads_started":1}
I20260924 21:10:38.157943 15789 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 21:10:38.158355 15789 catalog_manager.cc:1504] Loading token signing keys...
I20260924 21:10:38.162410 15789 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a18533bad310410ca2873fde2d8ff491: Generated new TSK 0
I20260924 21:10:38.162563 15789 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790284076800039-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790284076800039-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260924 21:10:38.186391 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAzNNjS/build/release/bin/kudu
/tmp/dist-test-taskAzNNjS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790284076800039-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790284076800039-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790284076800039-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790284076800039-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39885
--keytab_file=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790284076800039-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44143
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790284076800039-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790284076800039-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790284076800039-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 21:10:38.256183 15801 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 21:10:38.256342 15801 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 21:10:38.256361 15801 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 21:10:38.257238 15801 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 21:10:38.257272 15801 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 21:10:38.257293 15801 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 21:10:38.258436 15801 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44143
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790284076800039-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790284076800039-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790284076800039-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790284076800039-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39885
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15801
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790284076800039-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 19:49:06 UTC on ccycloud.quasar-vhqjoc.root.comops.site
build id 4670864
I20260924 21:10:38.258577 15801 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 21:10:38.258708 15801 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 21:10:38.262957 15807 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 21:10:38.263054 15801 server_base.cc:1034] running on GCE node
W20260924 21:10:38.262946 15806 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 21:10:38.262943 15809 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 21:10:38.263441 15801 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 21:10:38.263622 15801 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 21:10:38.264773 15801 hybrid_clock.cc:648] HybridClock initialized: now 1790284238264762 us; error 26 us; skew 500 ppm
Sep 24 21:10:38 dist-test-slave-c7tk krb5kdc[15694](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284238, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 21:10:38.267084 15801 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 21:10:38.267444 15801 webserver.cc:460] Webserver started at http://127.0.98.1:39809/ using document root <none> and password file <none>
I20260924 21:10:38.267660 15801 fs_manager.cc:359] Metadata directory not provided
I20260924 21:10:38.267719 15801 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 21:10:38.267851 15801 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 21:10:38.268693 15801 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790284076800039-392-0/minicluster-data/ts-0/data/instance:
uuid: "1d9aeb8d92ac439289b0193dee78ad72"
format_stamp: "Formatted at 2026-09-24 21:10:38 on dist-test-slave-c7tk"
server_key: "d72b83a7a68945cb826b8a2427e7a09f"
server_key_iv: "ac8be138b9e3bfd4dcf2a303f16243a0"
server_key_version: "encryptionkey@0"
I20260924 21:10:38.269008 15801 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790284076800039-392-0/minicluster-data/ts-0/wal/instance:
uuid: "1d9aeb8d92ac439289b0193dee78ad72"
format_stamp: "Formatted at 2026-09-24 21:10:38 on dist-test-slave-c7tk"
server_key: "d72b83a7a68945cb826b8a2427e7a09f"
server_key_iv: "ac8be138b9e3bfd4dcf2a303f16243a0"
server_key_version: "encryptionkey@0"
I20260924 21:10:38.270117 15801 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 21:10:38.270684 15816 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 21:10:38.270829 15801 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 21:10:38.270908 15801 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790284076800039-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790284076800039-392-0/minicluster-data/ts-0/wal
uuid: "1d9aeb8d92ac439289b0193dee78ad72"
format_stamp: "Formatted at 2026-09-24 21:10:38 on dist-test-slave-c7tk"
server_key: "d72b83a7a68945cb826b8a2427e7a09f"
server_key_iv: "ac8be138b9e3bfd4dcf2a303f16243a0"
server_key_version: "encryptionkey@0"
I20260924 21:10:38.270974 15801 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790284076800039-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790284076800039-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790284076800039-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 21:10:38.286374 15801 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 21:10:38.287814 15801 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 21:10:38.287950 15801 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 21:10:38.288156 15801 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 21:10:38.288395 15801 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 21:10:38.288440 15801 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 21:10:38.288467 15801 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 21:10:38.288491 15801 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 21:10:38.293372 15801 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:34291
I20260924 21:10:38.293443 15929 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:34291 every 8 connection(s)
I20260924 21:10:38.293717 15801 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790284076800039-392-0/minicluster-data/ts-0/data/info.pb
Sep 24 21:10:38 dist-test-slave-c7tk krb5kdc[15694](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284238, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 21:10:38.299918 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAzNNjS/build/release/bin/kudu as pid 15801
I20260924 21:10:38.300010 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790284076800039-392-0/minicluster-data/ts-0/wal/instance
I20260924 21:10:38.300220 392 external_mini_cluster.cc:1428] Setting key fd01a98d8ca36fe1a841a00e0dcd8ab5
I20260924 21:10:38.300981 15779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:10:38.294387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:43675 (local address 127.0.98.62:39885)
0924 21:10:38.294435 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0924 21:10:38.294438 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 21:10:38.294607 (+ 169us) server_negotiation.cc:407] Connection header received
0924 21:10:38.294929 (+ 322us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:10:38.294930 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:10:38.294939 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:10:38.294979 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:10:38.295681 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:10:38.296085 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:10:38.296894 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:10:38.297060 (+ 166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:10:38.297104 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:10:38.298802 (+ 1698us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 21:10:38.298803 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 21:10:38.298803 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 21:10:38.298810 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 21:10:38.299750 (+ 940us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:10:38.300173 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:10:38.300174 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:10:38.300174 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:10:38.300218 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:10:38.300457 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:10:38.300458 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:10:38.300458 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:10:38.300573 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 21:10:38.300630 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0924 21:10:38.300834 (+ 204us) server_negotiation.cc:299] Negotiation successful
0924 21:10:38.300904 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260924 21:10:38.301129 15932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:10:38.294379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39885 (local address 127.0.98.1:43675)
0924 21:10:38.294506 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0924 21:10:38.294532 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0924 21:10:38.294860 (+ 328us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 21:10:38.295024 (+ 164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 21:10:38.295027 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 21:10:38.295473 (+ 446us) client_negotiation.cc:189] Negotiated authn=SASL
0924 21:10:38.295618 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:10:38.295622 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:10:38.296162 (+ 540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:10:38.296163 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:10:38.296815 (+ 652us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:10:38.296816 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:10:38.297122 (+ 306us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:10:38.297123 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:10:38.297207 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:10:38.297575 (+ 368us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 21:10:38.297582 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0924 21:10:38.298734 (+ 1152us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 21:10:38.299832 (+ 1098us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 21:10:38.299838 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 21:10:38.299840 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 21:10:38.300105 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 21:10:38.300278 (+ 173us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 21:10:38.300279 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 21:10:38.300279 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 21:10:38.300394 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 21:10:38.300628 (+ 234us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 21:10:38.300632 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 21:10:38.300721 (+ 89us) client_negotiation.cc:769] Sending connection context
0924 21:10:38.300821 (+ 100us) client_negotiation.cc:240] Negotiation successful
0924 21:10:38.300947 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":69,"threads_started":1}
I20260924 21:10:38.302227 15930 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39885
I20260924 21:10:38.302325 15930 heartbeater.cc:464] Registering TS with master...
I20260924 21:10:38.302486 15930 heartbeater.cc:511] Master 127.0.98.62:39885 requested a full tablet report, sending...
I20260924 21:10:38.302939 15739 ts_manager.cc:194] Registered new tserver with Master: 1d9aeb8d92ac439289b0193dee78ad72 (127.0.98.1:34291)
I20260924 21:10:38.303798 15739 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:43675
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790284076800039-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790284076800039-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260924 21:10:38.321990 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAzNNjS/build/release/bin/kudu
/tmp/dist-test-taskAzNNjS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790284076800039-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790284076800039-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790284076800039-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790284076800039-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39885
--keytab_file=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790284076800039-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44143
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790284076800039-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790284076800039-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790284076800039-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 21:10:38.393693 15935 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 21:10:38.393904 15935 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 21:10:38.393929 15935 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 21:10:38.394913 15935 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 21:10:38.394946 15935 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 21:10:38.394990 15935 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 21:10:38.396230 15935 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44143
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790284076800039-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790284076800039-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790284076800039-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790284076800039-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39885
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15935
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790284076800039-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 19:49:06 UTC on ccycloud.quasar-vhqjoc.root.comops.site
build id 4670864
I20260924 21:10:38.396422 15935 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 21:10:38.396613 15935 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 21:10:38.400756 15941 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 21:10:38.400874 15935 server_base.cc:1034] running on GCE node
W20260924 21:10:38.400765 15943 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 21:10:38.400789 15940 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 21:10:38.401166 15935 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 21:10:38.401396 15935 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 21:10:38.402549 15935 hybrid_clock.cc:648] HybridClock initialized: now 1790284238402533 us; error 26 us; skew 500 ppm
Sep 24 21:10:38 dist-test-slave-c7tk krb5kdc[15694](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284238, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 21:10:38.404742 15935 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 21:10:38.405073 15935 webserver.cc:460] Webserver started at http://127.0.98.2:38699/ using document root <none> and password file <none>
I20260924 21:10:38.405272 15935 fs_manager.cc:359] Metadata directory not provided
I20260924 21:10:38.405314 15935 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 21:10:38.405467 15935 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 21:10:38.406462 15935 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790284076800039-392-0/minicluster-data/ts-1/data/instance:
uuid: "55193fd102fc483ab492f9b5a184b225"
format_stamp: "Formatted at 2026-09-24 21:10:38 on dist-test-slave-c7tk"
server_key: "010dc19283e82651e052ef7c67606224"
server_key_iv: "d271153449d1e8c7437aa1571c4a91a5"
server_key_version: "encryptionkey@0"
I20260924 21:10:38.406778 15935 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790284076800039-392-0/minicluster-data/ts-1/wal/instance:
uuid: "55193fd102fc483ab492f9b5a184b225"
format_stamp: "Formatted at 2026-09-24 21:10:38 on dist-test-slave-c7tk"
server_key: "010dc19283e82651e052ef7c67606224"
server_key_iv: "d271153449d1e8c7437aa1571c4a91a5"
server_key_version: "encryptionkey@0"
I20260924 21:10:38.407889 15935 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260924 21:10:38.408483 15950 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 21:10:38.408638 15935 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 21:10:38.408697 15935 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790284076800039-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790284076800039-392-0/minicluster-data/ts-1/wal
uuid: "55193fd102fc483ab492f9b5a184b225"
format_stamp: "Formatted at 2026-09-24 21:10:38 on dist-test-slave-c7tk"
server_key: "010dc19283e82651e052ef7c67606224"
server_key_iv: "d271153449d1e8c7437aa1571c4a91a5"
server_key_version: "encryptionkey@0"
I20260924 21:10:38.408771 15935 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790284076800039-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790284076800039-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790284076800039-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 21:10:38.420346 15935 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 21:10:38.421720 15935 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 21:10:38.421882 15935 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 21:10:38.422036 15935 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 21:10:38.422259 15935 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 21:10:38.422287 15935 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 21:10:38.422305 15935 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 21:10:38.422318 15935 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 21:10:38.427240 15935 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:38129
I20260924 21:10:38.427326 16063 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:38129 every 8 connection(s)
I20260924 21:10:38.427584 15935 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790284076800039-392-0/minicluster-data/ts-1/data/info.pb
Sep 24 21:10:38 dist-test-slave-c7tk krb5kdc[15694](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284238, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 21:10:38.435137 15779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:10:38.428222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:46839 (local address 127.0.98.62:39885)
0924 21:10:38.428267 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0924 21:10:38.428270 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 21:10:38.428454 (+ 184us) server_negotiation.cc:407] Connection header received
0924 21:10:38.428852 (+ 398us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:10:38.428853 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:10:38.428861 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:10:38.428903 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:10:38.429676 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:10:38.430136 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:10:38.431011 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:10:38.431169 (+ 158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:10:38.431217 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:10:38.433134 (+ 1917us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 21:10:38.433135 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 21:10:38.433135 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 21:10:38.433141 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 21:10:38.434249 (+ 1108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:10:38.434536 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:10:38.434537 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:10:38.434537 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:10:38.434578 (+ 41us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:10:38.434746 (+ 168us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:10:38.434746 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:10:38.434747 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:10:38.434830 (+ 83us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 21:10:38.434871 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0924 21:10:38.435036 (+ 165us) server_negotiation.cc:299] Negotiation successful
0924 21:10:38.435084 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260924 21:10:38.435316 16066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:10:38.428201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39885 (local address 127.0.98.2:46839)
0924 21:10:38.428343 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0924 21:10:38.428371 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0924 21:10:38.428776 (+ 405us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 21:10:38.428951 (+ 175us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 21:10:38.428953 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 21:10:38.429456 (+ 503us) client_negotiation.cc:189] Negotiated authn=SASL
0924 21:10:38.429599 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:10:38.429602 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:10:38.430199 (+ 597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:10:38.430199 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:10:38.430950 (+ 751us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:10:38.430953 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:10:38.431228 (+ 275us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:10:38.431229 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:10:38.431315 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:10:38.431816 (+ 501us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 21:10:38.431822 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0924 21:10:38.433055 (+ 1233us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 21:10:38.434306 (+ 1251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 21:10:38.434311 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 21:10:38.434313 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 21:10:38.434481 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 21:10:38.434627 (+ 146us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 21:10:38.434627 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 21:10:38.434628 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 21:10:38.434701 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 21:10:38.434876 (+ 175us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 21:10:38.434879 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 21:10:38.434961 (+ 82us) client_negotiation.cc:769] Sending connection context
0924 21:10:38.435022 (+ 61us) client_negotiation.cc:240] Negotiation successful
0924 21:10:38.435108 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":74,"threads_started":1}
I20260924 21:10:38.435765 16064 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39885
I20260924 21:10:38.435853 16064 heartbeater.cc:464] Registering TS with master...
I20260924 21:10:38.435901 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAzNNjS/build/release/bin/kudu as pid 15935
I20260924 21:10:38.436000 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790284076800039-392-0/minicluster-data/ts-1/wal/instance
I20260924 21:10:38.436022 16064 heartbeater.cc:511] Master 127.0.98.62:39885 requested a full tablet report, sending...
I20260924 21:10:38.436196 392 external_mini_cluster.cc:1428] Setting key 2b27ebb8a9c20c7bca78c5564d4a480e
I20260924 21:10:38.436342 15739 ts_manager.cc:194] Registered new tserver with Master: 55193fd102fc483ab492f9b5a184b225 (127.0.98.2:38129)
I20260924 21:10:38.436712 15739 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:46839
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790284076800039-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790284076800039-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260924 21:10:38.456429 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAzNNjS/build/release/bin/kudu
/tmp/dist-test-taskAzNNjS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790284076800039-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790284076800039-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790284076800039-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790284076800039-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39885
--keytab_file=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790284076800039-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44143
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790284076800039-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790284076800039-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790284076800039-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 21:10:38.524880 16069 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 21:10:38.525063 16069 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 21:10:38.525096 16069 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 21:10:38.526171 16069 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 21:10:38.526234 16069 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 21:10:38.526275 16069 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 21:10:38.527560 16069 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44143
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790284076800039-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790284076800039-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790284076800039-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790284076800039-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39885
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16069
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790284076800039-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 19:49:06 UTC on ccycloud.quasar-vhqjoc.root.comops.site
build id 4670864
I20260924 21:10:38.527783 16069 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 21:10:38.527983 16069 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 21:10:38.532310 16075 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 21:10:38.532344 16077 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 21:10:38.532326 16074 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 21:10:38.532497 16069 server_base.cc:1034] running on GCE node
I20260924 21:10:38.532773 16069 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 21:10:38.532969 16069 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 21:10:38.534129 16069 hybrid_clock.cc:648] HybridClock initialized: now 1790284238534115 us; error 28 us; skew 500 ppm
Sep 24 21:10:38 dist-test-slave-c7tk krb5kdc[15694](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284238, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 21:10:38.536254 16069 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 21:10:38.536628 16069 webserver.cc:460] Webserver started at http://127.0.98.3:44675/ using document root <none> and password file <none>
I20260924 21:10:38.536778 16069 fs_manager.cc:359] Metadata directory not provided
I20260924 21:10:38.536811 16069 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 21:10:38.536880 16069 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 21:10:38.537643 16069 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790284076800039-392-0/minicluster-data/ts-2/data/instance:
uuid: "e99c21581cf14320a2db35bd3ba7e8c6"
format_stamp: "Formatted at 2026-09-24 21:10:38 on dist-test-slave-c7tk"
server_key: "98c6f865964e69523d7f33892db0edea"
server_key_iv: "f396dd3f02cffa8e0103119e0154437a"
server_key_version: "encryptionkey@0"
I20260924 21:10:38.537945 16069 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790284076800039-392-0/minicluster-data/ts-2/wal/instance:
uuid: "e99c21581cf14320a2db35bd3ba7e8c6"
format_stamp: "Formatted at 2026-09-24 21:10:38 on dist-test-slave-c7tk"
server_key: "98c6f865964e69523d7f33892db0edea"
server_key_iv: "f396dd3f02cffa8e0103119e0154437a"
server_key_version: "encryptionkey@0"
I20260924 21:10:38.538972 16069 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 21:10:38.539618 16084 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 21:10:38.539774 16069 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 21:10:38.539856 16069 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790284076800039-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790284076800039-392-0/minicluster-data/ts-2/wal
uuid: "e99c21581cf14320a2db35bd3ba7e8c6"
format_stamp: "Formatted at 2026-09-24 21:10:38 on dist-test-slave-c7tk"
server_key: "98c6f865964e69523d7f33892db0edea"
server_key_iv: "f396dd3f02cffa8e0103119e0154437a"
server_key_version: "encryptionkey@0"
I20260924 21:10:38.539922 16069 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790284076800039-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790284076800039-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790284076800039-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 21:10:38.566883 16069 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 21:10:38.568471 16069 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 21:10:38.568562 16069 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 21:10:38.568682 16069 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 21:10:38.568872 16069 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 21:10:38.568897 16069 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 21:10:38.568914 16069 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 21:10:38.568925 16069 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 21:10:38.573994 16069 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:34643
I20260924 21:10:38.574059 16197 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:34643 every 8 connection(s)
I20260924 21:10:38.574299 16069 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790284076800039-392-0/minicluster-data/ts-2/data/info.pb
Sep 24 21:10:38 dist-test-slave-c7tk krb5kdc[15694](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284238, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 21:10:38.579967 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAzNNjS/build/release/bin/kudu as pid 16069
I20260924 21:10:38.580072 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790284076800039-392-0/minicluster-data/ts-2/wal/instance
I20260924 21:10:38.580269 392 external_mini_cluster.cc:1428] Setting key b2ecd24fbc644378175519a3079ac7c0
I20260924 21:10:38.582172 15779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:10:38.574888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:58329 (local address 127.0.98.62:39885)
0924 21:10:38.574926 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0924 21:10:38.574928 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 21:10:38.575095 (+ 167us) server_negotiation.cc:407] Connection header received
0924 21:10:38.575527 (+ 432us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:10:38.575528 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:10:38.575540 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:10:38.575596 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:10:38.576392 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:10:38.576818 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:10:38.577677 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:10:38.577921 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:10:38.577993 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:10:38.579712 (+ 1719us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 21:10:38.579713 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 21:10:38.579714 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 21:10:38.579720 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 21:10:38.581098 (+ 1378us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:10:38.581440 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:10:38.581441 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:10:38.581441 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:10:38.581488 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:10:38.581636 (+ 148us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:10:38.581636 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:10:38.581636 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:10:38.581719 (+ 83us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 21:10:38.581766 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0924 21:10:38.582037 (+ 271us) server_negotiation.cc:299] Negotiation successful
0924 21:10:38.582113 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260924 21:10:38.582297 16200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:10:38.574848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39885 (local address 127.0.98.3:58329)
0924 21:10:38.574965 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0924 21:10:38.575023 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0924 21:10:38.575432 (+ 409us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 21:10:38.575652 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 21:10:38.575655 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 21:10:38.576138 (+ 483us) client_negotiation.cc:189] Negotiated authn=SASL
0924 21:10:38.576302 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:10:38.576305 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:10:38.576898 (+ 593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:10:38.576899 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:10:38.577563 (+ 664us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:10:38.577564 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:10:38.578001 (+ 437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:10:38.578002 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:10:38.578105 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:10:38.578509 (+ 404us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 21:10:38.578516 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0924 21:10:38.579636 (+ 1120us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 21:10:38.581170 (+ 1534us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 21:10:38.581174 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 21:10:38.581175 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 21:10:38.581382 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 21:10:38.581524 (+ 142us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 21:10:38.581524 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 21:10:38.581524 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 21:10:38.581592 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 21:10:38.581757 (+ 165us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 21:10:38.581759 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 21:10:38.581932 (+ 173us) client_negotiation.cc:769] Sending connection context
0924 21:10:38.582008 (+ 76us) client_negotiation.cc:240] Negotiation successful
0924 21:10:38.582133 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":86,"thread_start_us":59,"threads_started":1}
I20260924 21:10:38.582873 16198 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39885
I20260924 21:10:38.582957 16198 heartbeater.cc:464] Registering TS with master...
I20260924 21:10:38.583129 16198 heartbeater.cc:511] Master 127.0.98.62:39885 requested a full tablet report, sending...
I20260924 21:10:38.583488 15739 ts_manager.cc:194] Registered new tserver with Master: e99c21581cf14320a2db35bd3ba7e8c6 (127.0.98.3:34643)
I20260924 21:10:38.583822 15739 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:58329
I20260924 21:10:38.591720 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 21:10:38.597854 15779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:10:38.593048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46312 (local address 127.0.98.62:39885)
0924 21:10:38.593127 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0924 21:10:38.593130 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 21:10:38.593265 (+ 135us) server_negotiation.cc:407] Connection header received
0924 21:10:38.593328 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:10:38.593329 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:10:38.593336 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:10:38.593378 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:10:38.593893 (+ 515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:10:38.594284 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:10:38.594860 (+ 576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:10:38.594999 (+ 139us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:10:38.595038 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:10:38.595948 (+ 910us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 21:10:38.595948 (+ 0us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 21:10:38.595948 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 21:10:38.595952 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 21:10:38.596915 (+ 963us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:10:38.597184 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:10:38.597185 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:10:38.597185 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:10:38.597242 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:10:38.597418 (+ 176us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:10:38.597419 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:10:38.597419 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:10:38.597523 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 21:10:38.597597 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0924 21:10:38.597703 (+ 106us) server_negotiation.cc:299] Negotiation successful
0924 21:10:38.597764 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260924 21:10:38.599148 15739 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46312:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 21:10:38.600627 15739 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 21:10:38.605912 16206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:10:38.602666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:34643 (local address 127.0.0.1:42788)
0924 21:10:38.602753 (+ 87us) negotiation.cc:107] Waiting for socket to connect
0924 21:10:38.602795 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 21:10:38.602848 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 21:10:38.603030 (+ 182us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 21:10:38.603033 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 21:10:38.603036 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 21:10:38.603095 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:10:38.603097 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:10:38.604189 (+ 1092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:10:38.604189 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:10:38.604892 (+ 703us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:10:38.604894 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:10:38.605644 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:10:38.605644 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:10:38.605718 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:10:38.605729 (+ 11us) client_negotiation.cc:769] Sending connection context
0924 21:10:38.605811 (+ 82us) client_negotiation.cc:240] Negotiation successful
0924 21:10:38.605856 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":68,"thread_start_us":50,"threads_started":1}
I20260924 21:10:38.605948 16207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:10:38.602726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42788 (local address 127.0.98.3:34643)
0924 21:10:38.602868 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0924 21:10:38.602872 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 21:10:38.602882 (+ 10us) server_negotiation.cc:407] Connection header received
0924 21:10:38.602920 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:10:38.602921 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:10:38.602932 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:10:38.603024 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 21:10:38.603165 (+ 141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:10:38.604096 (+ 931us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:10:38.604963 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:10:38.605534 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:10:38.605623 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:10:38.605668 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0924 21:10:38.605837 (+ 169us) server_negotiation.cc:299] Negotiation successful
0924 21:10:38.605872 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115,"thread_start_us":89,"threads_started":1}
I20260924 21:10:38.606563 16132 tablet_service.cc:1467] Processing CreateTablet for tablet 4fe70b55ce09492691ac635e09462401 (DEFAULT_TABLE table=test-table [id=8f52a32a3a8e4afda4b62b9791c2a98d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 21:10:38.606675 16209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:10:38.603016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38129 (local address 127.0.0.1:51314)
0924 21:10:38.603107 (+ 91us) negotiation.cc:107] Waiting for socket to connect
0924 21:10:38.603114 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0924 21:10:38.603149 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 21:10:38.603661 (+ 512us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 21:10:38.603662 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 21:10:38.603664 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 21:10:38.603704 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:10:38.603705 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:10:38.604847 (+ 1142us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:10:38.604848 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:10:38.605646 (+ 798us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:10:38.605647 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:10:38.606466 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:10:38.606467 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:10:38.606521 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:10:38.606524 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 21:10:38.606576 (+ 52us) client_negotiation.cc:240] Negotiation successful
0924 21:10:38.606610 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":79,"thread_start_us":60,"threads_started":1}
I20260924 21:10:38.606688 16211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:10:38.603313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51314 (local address 127.0.98.2:38129)
0924 21:10:38.603457 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0924 21:10:38.603461 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 21:10:38.603470 (+ 9us) server_negotiation.cc:407] Connection header received
0924 21:10:38.603506 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:10:38.603508 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:10:38.603522 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:10:38.603592 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 21:10:38.603775 (+ 183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:10:38.604745 (+ 970us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:10:38.605723 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:10:38.606369 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:10:38.606463 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:10:38.606514 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0924 21:10:38.606588 (+ 74us) server_negotiation.cc:299] Negotiation successful
0924 21:10:38.606621 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115,"thread_start_us":90,"threads_started":1}
I20260924 21:10:38.606864 16132 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4fe70b55ce09492691ac635e09462401. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 21:10:38.607231 15998 tablet_service.cc:1467] Processing CreateTablet for tablet 4fe70b55ce09492691ac635e09462401 (DEFAULT_TABLE table=test-table [id=8f52a32a3a8e4afda4b62b9791c2a98d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 21:10:38.607378 16208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:10:38.602850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34291 (local address 127.0.0.1:50460)
0924 21:10:38.602973 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0924 21:10:38.602981 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0924 21:10:38.603018 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 21:10:38.603738 (+ 720us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 21:10:38.603739 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 21:10:38.603741 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 21:10:38.603788 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:10:38.603789 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:10:38.604873 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:10:38.604874 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:10:38.605852 (+ 978us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:10:38.605854 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:10:38.607153 (+ 1299us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:10:38.607164 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:10:38.607219 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:10:38.607223 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 21:10:38.607291 (+ 68us) client_negotiation.cc:240] Negotiation successful
0924 21:10:38.607324 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":81,"threads_started":1}
I20260924 21:10:38.607429 16210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:10:38.603300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50460 (local address 127.0.98.1:34291)
0924 21:10:38.603583 (+ 283us) server_negotiation.cc:206] Beginning negotiation
0924 21:10:38.603586 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 21:10:38.603594 (+ 8us) server_negotiation.cc:407] Connection header received
0924 21:10:38.603628 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:10:38.603630 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:10:38.603643 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:10:38.603738 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 21:10:38.603845 (+ 107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:10:38.604771 (+ 926us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:10:38.605931 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:10:38.607073 (+ 1142us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:10:38.607145 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:10:38.607203 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0924 21:10:38.607303 (+ 100us) server_negotiation.cc:299] Negotiation successful
0924 21:10:38.607342 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":252,"thread_start_us":78,"threads_started":1}
I20260924 21:10:38.607530 15998 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4fe70b55ce09492691ac635e09462401. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 21:10:38.608028 15864 tablet_service.cc:1467] Processing CreateTablet for tablet 4fe70b55ce09492691ac635e09462401 (DEFAULT_TABLE table=test-table [id=8f52a32a3a8e4afda4b62b9791c2a98d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 21:10:38.608306 15864 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4fe70b55ce09492691ac635e09462401. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 21:10:38.611019 16214 tablet_bootstrap.cc:492] T 4fe70b55ce09492691ac635e09462401 P 1d9aeb8d92ac439289b0193dee78ad72: Bootstrap starting.
I20260924 21:10:38.611096 16213 tablet_bootstrap.cc:492] T 4fe70b55ce09492691ac635e09462401 P 55193fd102fc483ab492f9b5a184b225: Bootstrap starting.
I20260924 21:10:38.611575 16214 tablet_bootstrap.cc:654] T 4fe70b55ce09492691ac635e09462401 P 1d9aeb8d92ac439289b0193dee78ad72: Neither blocks nor log segments found. Creating new log.
I20260924 21:10:38.611594 16213 tablet_bootstrap.cc:654] T 4fe70b55ce09492691ac635e09462401 P 55193fd102fc483ab492f9b5a184b225: Neither blocks nor log segments found. Creating new log.
I20260924 21:10:38.612066 16214 log.cc:824] T 4fe70b55ce09492691ac635e09462401 P 1d9aeb8d92ac439289b0193dee78ad72: Log is configured to *not* fsync() on all Append() calls
I20260924 21:10:38.612066 16213 log.cc:824] T 4fe70b55ce09492691ac635e09462401 P 55193fd102fc483ab492f9b5a184b225: Log is configured to *not* fsync() on all Append() calls
I20260924 21:10:38.612418 16212 tablet_bootstrap.cc:492] T 4fe70b55ce09492691ac635e09462401 P e99c21581cf14320a2db35bd3ba7e8c6: Bootstrap starting.
I20260924 21:10:38.612774 16214 tablet_bootstrap.cc:492] T 4fe70b55ce09492691ac635e09462401 P 1d9aeb8d92ac439289b0193dee78ad72: No bootstrap required, opened a new log
I20260924 21:10:38.612813 16213 tablet_bootstrap.cc:492] T 4fe70b55ce09492691ac635e09462401 P 55193fd102fc483ab492f9b5a184b225: No bootstrap required, opened a new log
I20260924 21:10:38.612850 16214 ts_tablet_manager.cc:1397] T 4fe70b55ce09492691ac635e09462401 P 1d9aeb8d92ac439289b0193dee78ad72: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 21:10:38.612867 16213 ts_tablet_manager.cc:1397] T 4fe70b55ce09492691ac635e09462401 P 55193fd102fc483ab492f9b5a184b225: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 21:10:38.613068 16212 tablet_bootstrap.cc:654] T 4fe70b55ce09492691ac635e09462401 P e99c21581cf14320a2db35bd3ba7e8c6: Neither blocks nor log segments found. Creating new log.
I20260924 21:10:38.613619 16212 log.cc:824] T 4fe70b55ce09492691ac635e09462401 P e99c21581cf14320a2db35bd3ba7e8c6: Log is configured to *not* fsync() on all Append() calls
I20260924 21:10:38.614421 16212 tablet_bootstrap.cc:492] T 4fe70b55ce09492691ac635e09462401 P e99c21581cf14320a2db35bd3ba7e8c6: No bootstrap required, opened a new log
I20260924 21:10:38.614483 16212 ts_tablet_manager.cc:1397] T 4fe70b55ce09492691ac635e09462401 P e99c21581cf14320a2db35bd3ba7e8c6: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260924 21:10:38.614609 16214 raft_consensus.cc:348] T 4fe70b55ce09492691ac635e09462401 P 1d9aeb8d92ac439289b0193dee78ad72 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e99c21581cf14320a2db35bd3ba7e8c6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34643 } } peers { permanent_uuid: "1d9aeb8d92ac439289b0193dee78ad72" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34291 } } peers { permanent_uuid: "55193fd102fc483ab492f9b5a184b225" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38129 } }
I20260924 21:10:38.614781 16214 raft_consensus.cc:374] T 4fe70b55ce09492691ac635e09462401 P 1d9aeb8d92ac439289b0193dee78ad72 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 21:10:38.614821 16214 raft_consensus.cc:729] T 4fe70b55ce09492691ac635e09462401 P 1d9aeb8d92ac439289b0193dee78ad72 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d9aeb8d92ac439289b0193dee78ad72, State: Initialized, Role: FOLLOWER
I20260924 21:10:38.614938 16214 consensus_queue.cc:260] T 4fe70b55ce09492691ac635e09462401 P 1d9aeb8d92ac439289b0193dee78ad72 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e99c21581cf14320a2db35bd3ba7e8c6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34643 } } peers { permanent_uuid: "1d9aeb8d92ac439289b0193dee78ad72" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34291 } } peers { permanent_uuid: "55193fd102fc483ab492f9b5a184b225" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38129 } }
I20260924 21:10:38.615025 16213 raft_consensus.cc:348] T 4fe70b55ce09492691ac635e09462401 P 55193fd102fc483ab492f9b5a184b225 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e99c21581cf14320a2db35bd3ba7e8c6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34643 } } peers { permanent_uuid: "1d9aeb8d92ac439289b0193dee78ad72" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34291 } } peers { permanent_uuid: "55193fd102fc483ab492f9b5a184b225" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38129 } }
I20260924 21:10:38.615180 16213 raft_consensus.cc:374] T 4fe70b55ce09492691ac635e09462401 P 55193fd102fc483ab492f9b5a184b225 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 21:10:38.615192 16214 ts_tablet_manager.cc:1428] T 4fe70b55ce09492691ac635e09462401 P 1d9aeb8d92ac439289b0193dee78ad72: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 21:10:38.615218 16213 raft_consensus.cc:729] T 4fe70b55ce09492691ac635e09462401 P 55193fd102fc483ab492f9b5a184b225 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 55193fd102fc483ab492f9b5a184b225, State: Initialized, Role: FOLLOWER
I20260924 21:10:38.615288 15930 heartbeater.cc:503] Master 127.0.98.62:39885 was elected leader, sending a full tablet report...
I20260924 21:10:38.615360 16213 consensus_queue.cc:260] T 4fe70b55ce09492691ac635e09462401 P 55193fd102fc483ab492f9b5a184b225 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e99c21581cf14320a2db35bd3ba7e8c6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34643 } } peers { permanent_uuid: "1d9aeb8d92ac439289b0193dee78ad72" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34291 } } peers { permanent_uuid: "55193fd102fc483ab492f9b5a184b225" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38129 } }
I20260924 21:10:38.615607 16213 ts_tablet_manager.cc:1428] T 4fe70b55ce09492691ac635e09462401 P 55193fd102fc483ab492f9b5a184b225: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 21:10:38.615727 16064 heartbeater.cc:503] Master 127.0.98.62:39885 was elected leader, sending a full tablet report...
I20260924 21:10:38.616291 16212 raft_consensus.cc:348] T 4fe70b55ce09492691ac635e09462401 P e99c21581cf14320a2db35bd3ba7e8c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e99c21581cf14320a2db35bd3ba7e8c6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34643 } } peers { permanent_uuid: "1d9aeb8d92ac439289b0193dee78ad72" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34291 } } peers { permanent_uuid: "55193fd102fc483ab492f9b5a184b225" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38129 } }
I20260924 21:10:38.616458 16212 raft_consensus.cc:374] T 4fe70b55ce09492691ac635e09462401 P e99c21581cf14320a2db35bd3ba7e8c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 21:10:38.616501 16212 raft_consensus.cc:729] T 4fe70b55ce09492691ac635e09462401 P e99c21581cf14320a2db35bd3ba7e8c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e99c21581cf14320a2db35bd3ba7e8c6, State: Initialized, Role: FOLLOWER
I20260924 21:10:38.616597 16212 consensus_queue.cc:260] T 4fe70b55ce09492691ac635e09462401 P e99c21581cf14320a2db35bd3ba7e8c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e99c21581cf14320a2db35bd3ba7e8c6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34643 } } peers { permanent_uuid: "1d9aeb8d92ac439289b0193dee78ad72" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34291 } } peers { permanent_uuid: "55193fd102fc483ab492f9b5a184b225" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38129 } }
I20260924 21:10:38.616838 16212 ts_tablet_manager.cc:1428] T 4fe70b55ce09492691ac635e09462401 P e99c21581cf14320a2db35bd3ba7e8c6: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260924 21:10:38.616878 16198 heartbeater.cc:503] Master 127.0.98.62:39885 was elected leader, sending a full tablet report...
W20260924 21:10:38.678112 16065 tablet.cc:2369] T 4fe70b55ce09492691ac635e09462401 P 55193fd102fc483ab492f9b5a184b225: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 21:10:38.743148 16220 raft_consensus.cc:482] T 4fe70b55ce09492691ac635e09462401 P e99c21581cf14320a2db35bd3ba7e8c6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 21:10:38.743300 16220 raft_consensus.cc:504] T 4fe70b55ce09492691ac635e09462401 P e99c21581cf14320a2db35bd3ba7e8c6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e99c21581cf14320a2db35bd3ba7e8c6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34643 } } peers { permanent_uuid: "1d9aeb8d92ac439289b0193dee78ad72" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34291 } } peers { permanent_uuid: "55193fd102fc483ab492f9b5a184b225" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38129 } }
I20260924 21:10:38.743683 16220 leader_election.cc:288] T 4fe70b55ce09492691ac635e09462401 P e99c21581cf14320a2db35bd3ba7e8c6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1d9aeb8d92ac439289b0193dee78ad72 (127.0.98.1:34291), 55193fd102fc483ab492f9b5a184b225 (127.0.98.2:38129)
I20260924 21:10:38.746811 16211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:10:38.744038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:52621 (local address 127.0.98.2:38129)
0924 21:10:38.744078 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0924 21:10:38.744080 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 21:10:38.744235 (+ 155us) server_negotiation.cc:407] Connection header received
0924 21:10:38.744320 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:10:38.744321 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:10:38.744330 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:10:38.744388 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 21:10:38.744538 (+ 150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:10:38.745014 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:10:38.745938 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:10:38.746479 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:10:38.746552 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:10:38.746563 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0924 21:10:38.746713 (+ 150us) server_negotiation.cc:299] Negotiation successful
0924 21:10:38.746747 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260924 21:10:38.746834 16221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:10:38.743992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38129 (local address 127.0.98.3:52621)
0924 21:10:38.744167 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0924 21:10:38.744174 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0924 21:10:38.744244 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 21:10:38.744391 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 21:10:38.744392 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 21:10:38.744393 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 21:10:38.744463 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:10:38.744464 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:10:38.745092 (+ 628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:10:38.745093 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:10:38.745850 (+ 757us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:10:38.745851 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:10:38.746554 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:10:38.746555 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:10:38.746626 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:10:38.746629 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 21:10:38.746707 (+ 78us) client_negotiation.cc:240] Negotiation successful
0924 21:10:38.746746 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":110,"threads_started":1}
I20260924 21:10:38.747047 16200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:10:38.743869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34291 (local address 127.0.98.3:40523)
0924 21:10:38.743942 (+ 73us) negotiation.cc:107] Waiting for socket to connect
0924 21:10:38.743985 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 21:10:38.744049 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 21:10:38.744241 (+ 192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 21:10:38.744242 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 21:10:38.744243 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 21:10:38.744323 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:10:38.744324 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:10:38.745017 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:10:38.745018 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:10:38.745924 (+ 906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:10:38.745926 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:10:38.746753 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:10:38.746754 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:10:38.746843 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:10:38.746846 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 21:10:38.746919 (+ 73us) client_negotiation.cc:240] Negotiation successful
0924 21:10:38.746966 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49}
I20260924 21:10:38.747115 16018 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fe70b55ce09492691ac635e09462401" candidate_uuid: "e99c21581cf14320a2db35bd3ba7e8c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "55193fd102fc483ab492f9b5a184b225" is_pre_election: true
I20260924 21:10:38.747277 16018 raft_consensus.cc:2454] T 4fe70b55ce09492691ac635e09462401 P 55193fd102fc483ab492f9b5a184b225 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e99c21581cf14320a2db35bd3ba7e8c6 in term 0.
I20260924 21:10:38.747352 16210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:10:38.743940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:40523 (local address 127.0.98.1:34291)
0924 21:10:38.744021 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0924 21:10:38.744023 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 21:10:38.744035 (+ 12us) server_negotiation.cc:407] Connection header received
0924 21:10:38.744144 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:10:38.744146 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:10:38.744166 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:10:38.744230 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 21:10:38.744403 (+ 173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:10:38.744914 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:10:38.746014 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:10:38.746652 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:10:38.746754 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:10:38.746764 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0924 21:10:38.747239 (+ 475us) server_negotiation.cc:299] Negotiation successful
0924 21:10:38.747280 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260924 21:10:38.747498 16086 leader_election.cc:302] T 4fe70b55ce09492691ac635e09462401 P e99c21581cf14320a2db35bd3ba7e8c6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 55193fd102fc483ab492f9b5a184b225, e99c21581cf14320a2db35bd3ba7e8c6; no voters:
I20260924 21:10:38.747645 16220 raft_consensus.cc:2781] T 4fe70b55ce09492691ac635e09462401 P e99c21581cf14320a2db35bd3ba7e8c6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 21:10:38.747704 16220 raft_consensus.cc:482] T 4fe70b55ce09492691ac635e09462401 P e99c21581cf14320a2db35bd3ba7e8c6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 21:10:38.747730 16220 raft_consensus.cc:3037] T 4fe70b55ce09492691ac635e09462401 P e99c21581cf14320a2db35bd3ba7e8c6 [term 0 FOLLOWER]: Advancing to term 1
I20260924 21:10:38.747900 15884 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fe70b55ce09492691ac635e09462401" candidate_uuid: "e99c21581cf14320a2db35bd3ba7e8c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d9aeb8d92ac439289b0193dee78ad72" is_pre_election: true
I20260924 21:10:38.748057 15884 raft_consensus.cc:2454] T 4fe70b55ce09492691ac635e09462401 P 1d9aeb8d92ac439289b0193dee78ad72 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e99c21581cf14320a2db35bd3ba7e8c6 in term 0.
I20260924 21:10:38.748598 16220 raft_consensus.cc:504] T 4fe70b55ce09492691ac635e09462401 P e99c21581cf14320a2db35bd3ba7e8c6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e99c21581cf14320a2db35bd3ba7e8c6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34643 } } peers { permanent_uuid: "1d9aeb8d92ac439289b0193dee78ad72" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34291 } } peers { permanent_uuid: "55193fd102fc483ab492f9b5a184b225" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38129 } }
I20260924 21:10:38.748713 16220 leader_election.cc:288] T 4fe70b55ce09492691ac635e09462401 P e99c21581cf14320a2db35bd3ba7e8c6 [CANDIDATE]: Term 1 election: Requested vote from peers 1d9aeb8d92ac439289b0193dee78ad72 (127.0.98.1:34291), 55193fd102fc483ab492f9b5a184b225 (127.0.98.2:38129)
I20260924 21:10:38.748852 15884 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fe70b55ce09492691ac635e09462401" candidate_uuid: "e99c21581cf14320a2db35bd3ba7e8c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d9aeb8d92ac439289b0193dee78ad72"
I20260924 21:10:38.748871 16018 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fe70b55ce09492691ac635e09462401" candidate_uuid: "e99c21581cf14320a2db35bd3ba7e8c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "55193fd102fc483ab492f9b5a184b225"
I20260924 21:10:38.748934 15884 raft_consensus.cc:3037] T 4fe70b55ce09492691ac635e09462401 P 1d9aeb8d92ac439289b0193dee78ad72 [term 0 FOLLOWER]: Advancing to term 1
I20260924 21:10:38.748958 16018 raft_consensus.cc:3037] T 4fe70b55ce09492691ac635e09462401 P 55193fd102fc483ab492f9b5a184b225 [term 0 FOLLOWER]: Advancing to term 1
I20260924 21:10:38.749679 16018 raft_consensus.cc:2454] T 4fe70b55ce09492691ac635e09462401 P 55193fd102fc483ab492f9b5a184b225 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e99c21581cf14320a2db35bd3ba7e8c6 in term 1.
I20260924 21:10:38.749688 15884 raft_consensus.cc:2454] T 4fe70b55ce09492691ac635e09462401 P 1d9aeb8d92ac439289b0193dee78ad72 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e99c21581cf14320a2db35bd3ba7e8c6 in term 1.
I20260924 21:10:38.749861 16086 leader_election.cc:302] T 4fe70b55ce09492691ac635e09462401 P e99c21581cf14320a2db35bd3ba7e8c6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 55193fd102fc483ab492f9b5a184b225, e99c21581cf14320a2db35bd3ba7e8c6; no voters:
I20260924 21:10:38.749945 16220 raft_consensus.cc:2781] T 4fe70b55ce09492691ac635e09462401 P e99c21581cf14320a2db35bd3ba7e8c6 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 21:10:38.750100 16220 raft_consensus.cc:686] T 4fe70b55ce09492691ac635e09462401 P e99c21581cf14320a2db35bd3ba7e8c6 [term 1 LEADER]: Becoming Leader. State: Replica: e99c21581cf14320a2db35bd3ba7e8c6, State: Running, Role: LEADER
I20260924 21:10:38.750197 16220 consensus_queue.cc:237] T 4fe70b55ce09492691ac635e09462401 P e99c21581cf14320a2db35bd3ba7e8c6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e99c21581cf14320a2db35bd3ba7e8c6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34643 } } peers { permanent_uuid: "1d9aeb8d92ac439289b0193dee78ad72" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34291 } } peers { permanent_uuid: "55193fd102fc483ab492f9b5a184b225" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38129 } }
I20260924 21:10:38.751076 15739 catalog_manager.cc:5495] T 4fe70b55ce09492691ac635e09462401 P e99c21581cf14320a2db35bd3ba7e8c6 reported cstate change: term changed from 0 to 1, leader changed from <none> to e99c21581cf14320a2db35bd3ba7e8c6 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "e99c21581cf14320a2db35bd3ba7e8c6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e99c21581cf14320a2db35bd3ba7e8c6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34643 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1d9aeb8d92ac439289b0193dee78ad72" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34291 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "55193fd102fc483ab492f9b5a184b225" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38129 } health_report { overall_health: UNKNOWN } } }
I20260924 21:10:38.785724 16207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:10:38.783940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42804 (local address 127.0.98.3:34643)
0924 21:10:38.784002 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0924 21:10:38.784003 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 21:10:38.784011 (+ 8us) server_negotiation.cc:407] Connection header received
0924 21:10:38.784034 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:10:38.784035 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:10:38.784047 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:10:38.784118 (+ 71us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 21:10:38.784230 (+ 112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:10:38.784624 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:10:38.785193 (+ 569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:10:38.785343 (+ 150us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:10:38.785383 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:10:38.785495 (+ 112us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 21:10:38.785567 (+ 72us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 21:10:38.785615 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0924 21:10:38.785639 (+ 24us) server_negotiation.cc:299] Negotiation successful
0924 21:10:38.785652 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260924 21:10:38.787155 16018 raft_consensus.cc:1261] T 4fe70b55ce09492691ac635e09462401 P 55193fd102fc483ab492f9b5a184b225 [term 1 FOLLOWER]: Refusing update from remote peer e99c21581cf14320a2db35bd3ba7e8c6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 21:10:38.787158 15884 raft_consensus.cc:1261] T 4fe70b55ce09492691ac635e09462401 P 1d9aeb8d92ac439289b0193dee78ad72 [term 1 FOLLOWER]: Refusing update from remote peer e99c21581cf14320a2db35bd3ba7e8c6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 21:10:38.787470 16222 consensus_queue.cc:1035] T 4fe70b55ce09492691ac635e09462401 P e99c21581cf14320a2db35bd3ba7e8c6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "55193fd102fc483ab492f9b5a184b225" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38129 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 21:10:38.787575 16220 consensus_queue.cc:1035] T 4fe70b55ce09492691ac635e09462401 P e99c21581cf14320a2db35bd3ba7e8c6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d9aeb8d92ac439289b0193dee78ad72" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 21:10:38.789602 16227 mvcc.cc:205] Tried to move back new op lower bound from 7333004242070142976 to 7333004241921597440. Current Snapshot: MvccSnapshot[applied={T|T < 7333004242070142976}]
I20260924 21:10:38.789940 16225 mvcc.cc:205] Tried to move back new op lower bound from 7333004242070142976 to 7333004241921597440. Current Snapshot: MvccSnapshot[applied={T|T < 7333004242070142976}]
I20260924 21:10:38.790427 16226 mvcc.cc:205] Tried to move back new op lower bound from 7333004242070142976 to 7333004241921597440. Current Snapshot: MvccSnapshot[applied={T|T < 7333004242070142976}]
I20260924 21:10:38.791853 15739 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46312:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 21:10:38.791931 15739 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46312:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 21:10:38.792593 15739 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a18533bad310410ca2873fde2d8ff491: Sending DeleteTablet for 3 replicas of tablet 4fe70b55ce09492691ac635e09462401
I20260924 21:10:38.792959 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAzNNjS/build/release/bin/kudu with pid 15801
I20260924 21:10:38.792963 16132 tablet_service.cc:1514] Processing DeleteTablet for tablet 4fe70b55ce09492691ac635e09462401 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 21:10:38 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:42788
I20260924 21:10:38.792970 15998 tablet_service.cc:1514] Processing DeleteTablet for tablet 4fe70b55ce09492691ac635e09462401 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 21:10:38 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:51314
I20260924 21:10:38.793748 16235 tablet_replica.cc:330] T 4fe70b55ce09492691ac635e09462401 P e99c21581cf14320a2db35bd3ba7e8c6: stopping tablet replica
I20260924 21:10:38.793859 16235 raft_consensus.cc:2229] T 4fe70b55ce09492691ac635e09462401 P e99c21581cf14320a2db35bd3ba7e8c6 [term 1 LEADER]: Raft consensus shutting down.
I20260924 21:10:38.793933 16235 raft_consensus.cc:2258] T 4fe70b55ce09492691ac635e09462401 P e99c21581cf14320a2db35bd3ba7e8c6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 21:10:38.794335 16235 ts_tablet_manager.cc:1905] T 4fe70b55ce09492691ac635e09462401 P e99c21581cf14320a2db35bd3ba7e8c6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 21:10:38.796861 16235 ts_tablet_manager.cc:1918] T 4fe70b55ce09492691ac635e09462401 P e99c21581cf14320a2db35bd3ba7e8c6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 21:10:38.796926 16235 log.cc:1196] T 4fe70b55ce09492691ac635e09462401 P e99c21581cf14320a2db35bd3ba7e8c6: Deleting WAL directory at /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790284076800039-392-0/minicluster-data/ts-2/wal/wals/4fe70b55ce09492691ac635e09462401
I20260924 21:10:38.797173 16235 ts_tablet_manager.cc:1939] T 4fe70b55ce09492691ac635e09462401 P e99c21581cf14320a2db35bd3ba7e8c6: Deleting consensus metadata
I20260924 21:10:38.797523 15724 catalog_manager.cc:4841] TS e99c21581cf14320a2db35bd3ba7e8c6 (127.0.98.3:34643): tablet 4fe70b55ce09492691ac635e09462401 (table test-table [id=8f52a32a3a8e4afda4b62b9791c2a98d]) successfully deleted
W20260924 21:10:38.799134 15724 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:34291 (error 108)
I20260924 21:10:38.799538 16208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:10:38.799423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34291 (local address 127.0.0.1:50464)
0924 21:10:38.799477 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0924 21:10:38.799508 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:34291: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260924 21:10:38.799606 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAzNNjS/build/release/bin/kudu with pid 15935
W20260924 21:10:38.805327 15724 catalog_manager.cc:4584] TS 1d9aeb8d92ac439289b0193dee78ad72 (127.0.98.1:34291): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 4fe70b55ce09492691ac635e09462401: Network error: Client connection negotiation failed: client connection to 127.0.98.1:34291: connect: Connection refused (error 111)
I20260924 21:10:38.806039 16208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:10:38.805945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38129 (local address 127.0.0.1:51318)
0924 21:10:38.805978 (+ 33us) negotiation.cc:107] Waiting for socket to connect
0924 21:10:38.806005 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:38129: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":18}
I20260924 21:10:38.806052 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAzNNjS/build/release/bin/kudu with pid 16069
I20260924 21:10:38.810444 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAzNNjS/build/release/bin/kudu with pid 15707
I20260924 21:10:58.815240 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAzNNjS/build/release/bin/kudu
/tmp/dist-test-taskAzNNjS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790284076800039-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790284076800039-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790284076800039-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790284076800039-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39885
--webserver_interface=127.0.98.62
--webserver_port=38395
--keytab_file=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790284076800039-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44143
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39885
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790284076800039-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790284076800039-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790284076800039-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 21:10:58.888617 16238 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 21:10:58.888772 16238 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 21:10:58.888790 16238 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 21:10:58.889779 16238 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260924 21:10:58.889859 16238 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260924 21:10:58.889876 16238 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260924 21:10:58.889890 16238 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 21:10:58.889915 16238 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 21:10:58.889931 16238 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 21:10:58.889942 16238 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 21:10:58.889956 16238 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 21:10:58.891130 16238 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44143
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790284076800039-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790284076800039-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39885
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790284076800039-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39885
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790284076800039-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=38395
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16238
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790284076800039-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 19:49:06 UTC on ccycloud.quasar-vhqjoc.root.comops.site
build id 4670864
I20260924 21:10:58.891285 16238 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 21:10:58.891418 16238 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 21:10:58.899037 16245 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 21:10:58.899037 16247 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 21:10:58.899109 16244 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 21:10:58.899220 16238 server_base.cc:1034] running on GCE node
I20260924 21:10:58.899434 16238 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 21:10:58.899647 16238 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 21:10:58.900779 16238 hybrid_clock.cc:648] HybridClock initialized: now 1790284258900768 us; error 24 us; skew 500 ppm
Sep 24 21:10:58 dist-test-slave-c7tk krb5kdc[15694](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284258, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 21:10:58.906277 16238 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 21:10:58.906657 16238 webserver.cc:460] Webserver started at http://127.0.98.62:38395/ using document root <none> and password file <none>
I20260924 21:10:58.906888 16238 fs_manager.cc:359] Metadata directory not provided
I20260924 21:10:58.906973 16238 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 21:10:58.908082 16238 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 21:10:58.908630 16254 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 21:10:58.908795 16238 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 21:10:58.908888 16238 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790284076800039-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790284076800039-392-0/minicluster-data/master-0/wal
uuid: "a18533bad310410ca2873fde2d8ff491"
format_stamp: "Formatted at 2026-09-24 21:10:38 on dist-test-slave-c7tk"
server_key: "7a801cb9420e22bcf66f976e7e25d2fd"
server_key_iv: "db9a9a6f8c42619d34d226e73332aed5"
server_key_version: "encryptionkey@0"
I20260924 21:10:58.909204 16238 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790284076800039-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790284076800039-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790284076800039-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 21:10:58.937486 16238 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 21:10:58.939119 16238 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 21:10:58.939255 16238 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 21:10:58.942938 16238 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39885
I20260924 21:10:58.942970 16306 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39885 every 8 connection(s)
I20260924 21:10:58.943305 16238 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790284076800039-392-0/minicluster-data/master-0/data/info.pb
I20260924 21:10:58.944643 16307 sys_catalog.cc:263] Verifying existing consensus state
I20260924 21:10:58.945281 16307 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a18533bad310410ca2873fde2d8ff491: Bootstrap starting.
I20260924 21:10:58.949981 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAzNNjS/build/release/bin/kudu as pid 16238
I20260924 21:10:58.950289 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAzNNjS/build/release/bin/kudu
/tmp/dist-test-taskAzNNjS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790284076800039-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790284076800039-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790284076800039-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790284076800039-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:34291
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:39885
--webserver_port=39809
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790284076800039-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44143
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790284076800039-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790284076800039-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790284076800039-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 21:10:58.953223 16307 log.cc:824] T 00000000000000000000000000000000 P a18533bad310410ca2873fde2d8ff491: Log is configured to *not* fsync() on all Append() calls
I20260924 21:10:58.955060 16307 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a18533bad310410ca2873fde2d8ff491: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 21:10:58.955338 16307 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a18533bad310410ca2873fde2d8ff491: Bootstrap complete.
I20260924 21:10:58.957506 16307 raft_consensus.cc:348] T 00000000000000000000000000000000 P a18533bad310410ca2873fde2d8ff491 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a18533bad310410ca2873fde2d8ff491" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39885 } }
I20260924 21:10:58.957741 16307 raft_consensus.cc:729] T 00000000000000000000000000000000 P a18533bad310410ca2873fde2d8ff491 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a18533bad310410ca2873fde2d8ff491, State: Initialized, Role: FOLLOWER
I20260924 21:10:58.957955 16307 consensus_queue.cc:260] T 00000000000000000000000000000000 P a18533bad310410ca2873fde2d8ff491 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a18533bad310410ca2873fde2d8ff491" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39885 } }
I20260924 21:10:58.958051 16307 raft_consensus.cc:388] T 00000000000000000000000000000000 P a18533bad310410ca2873fde2d8ff491 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 21:10:58.958107 16307 raft_consensus.cc:482] T 00000000000000000000000000000000 P a18533bad310410ca2873fde2d8ff491 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 21:10:58.958153 16307 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a18533bad310410ca2873fde2d8ff491 [term 1 FOLLOWER]: Advancing to term 2
I20260924 21:10:58.959074 16307 raft_consensus.cc:504] T 00000000000000000000000000000000 P a18533bad310410ca2873fde2d8ff491 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a18533bad310410ca2873fde2d8ff491" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39885 } }
I20260924 21:10:58.959187 16307 leader_election.cc:302] T 00000000000000000000000000000000 P a18533bad310410ca2873fde2d8ff491 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a18533bad310410ca2873fde2d8ff491; no voters:
I20260924 21:10:58.959383 16307 leader_election.cc:288] T 00000000000000000000000000000000 P a18533bad310410ca2873fde2d8ff491 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260924 21:10:58.959414 16311 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a18533bad310410ca2873fde2d8ff491 [term 2 FOLLOWER]: Leader election won for term 2
I20260924 21:10:58.959520 16311 raft_consensus.cc:686] T 00000000000000000000000000000000 P a18533bad310410ca2873fde2d8ff491 [term 2 LEADER]: Becoming Leader. State: Replica: a18533bad310410ca2873fde2d8ff491, State: Running, Role: LEADER
I20260924 21:10:58.959619 16307 sys_catalog.cc:564] T 00000000000000000000000000000000 P a18533bad310410ca2873fde2d8ff491 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 21:10:58.959597 16311 consensus_queue.cc:237] T 00000000000000000000000000000000 P a18533bad310410ca2873fde2d8ff491 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a18533bad310410ca2873fde2d8ff491" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39885 } }
I20260924 21:10:58.959870 16311 sys_catalog.cc:455] T 00000000000000000000000000000000 P a18533bad310410ca2873fde2d8ff491 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a18533bad310410ca2873fde2d8ff491. Latest consensus state: current_term: 2 leader_uuid: "a18533bad310410ca2873fde2d8ff491" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a18533bad310410ca2873fde2d8ff491" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39885 } } }
I20260924 21:10:58.959934 16311 sys_catalog.cc:458] T 00000000000000000000000000000000 P a18533bad310410ca2873fde2d8ff491 [sys.catalog]: This master's current role is: LEADER
I20260924 21:10:58.960083 16312 sys_catalog.cc:455] T 00000000000000000000000000000000 P a18533bad310410ca2873fde2d8ff491 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "a18533bad310410ca2873fde2d8ff491" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a18533bad310410ca2873fde2d8ff491" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39885 } } }
I20260924 21:10:58.960209 16312 sys_catalog.cc:458] T 00000000000000000000000000000000 P a18533bad310410ca2873fde2d8ff491 [sys.catalog]: This master's current role is: LEADER
I20260924 21:10:58.960420 16315 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 21:10:58.961190 16315 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 21:10:58.961494 16315 catalog_manager.cc:1259] Loaded cluster ID: 00ea91f509a845098739aef3b1bf5fff
I20260924 21:10:58.961531 16315 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 21:10:58.962651 16315 catalog_manager.cc:1504] Loading token signing keys...
I20260924 21:10:58.962805 16315 catalog_manager.cc:5879] T 00000000000000000000000000000000 P a18533bad310410ca2873fde2d8ff491: Loaded TSK: 0
I20260924 21:10:58.972615 16315 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a18533bad310410ca2873fde2d8ff491: Generated new TSK 1
I20260924 21:10:58.972681 16315 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260924 21:10:59.028278 16309 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 21:10:59.028438 16309 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 21:10:59.028460 16309 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 21:10:59.029579 16309 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 21:10:59.029614 16309 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 21:10:59.029639 16309 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 21:10:59.030953 16309 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44143
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790284076800039-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790284076800039-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790284076800039-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:34291
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790284076800039-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=39809
--tserver_master_addrs=127.0.98.62:39885
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16309
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790284076800039-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 19:49:06 UTC on ccycloud.quasar-vhqjoc.root.comops.site
build id 4670864
I20260924 21:10:59.031149 16309 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 21:10:59.031356 16309 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 21:10:59.035879 16335 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 21:10:59.035887 16334 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 21:10:59.036053 16309 server_base.cc:1034] running on GCE node
W20260924 21:10:59.035887 16337 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 21:10:59.036285 16309 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 21:10:59.036521 16309 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 21:10:59.037672 16309 hybrid_clock.cc:648] HybridClock initialized: now 1790284259037662 us; error 24 us; skew 500 ppm
Sep 24 21:10:59 dist-test-slave-c7tk krb5kdc[15694](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284259, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 21:10:59.039983 16309 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 21:10:59.040436 16309 webserver.cc:460] Webserver started at http://127.0.98.1:39809/ using document root <none> and password file <none>
I20260924 21:10:59.040665 16309 fs_manager.cc:359] Metadata directory not provided
I20260924 21:10:59.040736 16309 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 21:10:59.041960 16309 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 21:10:59.042616 16344 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 21:10:59.042790 16309 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260924 21:10:59.042878 16309 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790284076800039-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790284076800039-392-0/minicluster-data/ts-0/wal
uuid: "1d9aeb8d92ac439289b0193dee78ad72"
format_stamp: "Formatted at 2026-09-24 21:10:38 on dist-test-slave-c7tk"
server_key: "d72b83a7a68945cb826b8a2427e7a09f"
server_key_iv: "ac8be138b9e3bfd4dcf2a303f16243a0"
server_key_version: "encryptionkey@0"
I20260924 21:10:59.043157 16309 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790284076800039-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790284076800039-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790284076800039-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 21:10:59.050436 16309 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 21:10:59.052299 16309 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 21:10:59.052446 16309 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 21:10:59.052672 16309 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 21:10:59.053089 16351 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260924 21:10:59.054421 16309 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260924 21:10:59.054474 16309 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260924 21:10:59.054497 16309 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260924 21:10:59.055069 16309 ts_tablet_manager.cc:610] Registered 1 tablets
I20260924 21:10:59.055112 16309 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260924 21:10:59.055150 16351 tablet_bootstrap.cc:492] T 4fe70b55ce09492691ac635e09462401 P 1d9aeb8d92ac439289b0193dee78ad72: Bootstrap starting.
I20260924 21:10:59.061996 16309 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:34291
I20260924 21:10:59.062110 16458 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:34291 every 8 connection(s)
I20260924 21:10:59.062433 16309 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790284076800039-392-0/minicluster-data/ts-0/data/info.pb
I20260924 21:10:59.064867 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAzNNjS/build/release/bin/kudu as pid 16309
I20260924 21:10:59.065155 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAzNNjS/build/release/bin/kudu
/tmp/dist-test-taskAzNNjS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790284076800039-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790284076800039-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790284076800039-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790284076800039-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:38129
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:39885
--webserver_port=38699
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790284076800039-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44143
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790284076800039-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790284076800039-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790284076800039-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 21:10:59.065442 16351 log.cc:824] T 4fe70b55ce09492691ac635e09462401 P 1d9aeb8d92ac439289b0193dee78ad72: Log is configured to *not* fsync() on all Append() calls
I20260924 21:10:59.067719 16351 tablet_bootstrap.cc:492] T 4fe70b55ce09492691ac635e09462401 P 1d9aeb8d92ac439289b0193dee78ad72: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 21:10:59.067989 16351 tablet_bootstrap.cc:492] T 4fe70b55ce09492691ac635e09462401 P 1d9aeb8d92ac439289b0193dee78ad72: Bootstrap complete.
I20260924 21:10:59.069005 16351 ts_tablet_manager.cc:1397] T 4fe70b55ce09492691ac635e09462401 P 1d9aeb8d92ac439289b0193dee78ad72: Time spent bootstrapping tablet: real 0.014s user 0.006s sys 0.004s
I20260924 21:10:59.070112 16351 raft_consensus.cc:348] T 4fe70b55ce09492691ac635e09462401 P 1d9aeb8d92ac439289b0193dee78ad72 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e99c21581cf14320a2db35bd3ba7e8c6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34643 } } peers { permanent_uuid: "1d9aeb8d92ac439289b0193dee78ad72" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34291 } } peers { permanent_uuid: "55193fd102fc483ab492f9b5a184b225" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38129 } }
I20260924 21:10:59.070428 16351 raft_consensus.cc:729] T 4fe70b55ce09492691ac635e09462401 P 1d9aeb8d92ac439289b0193dee78ad72 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d9aeb8d92ac439289b0193dee78ad72, State: Initialized, Role: FOLLOWER
I20260924 21:10:59.070564 16351 consensus_queue.cc:260] T 4fe70b55ce09492691ac635e09462401 P 1d9aeb8d92ac439289b0193dee78ad72 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e99c21581cf14320a2db35bd3ba7e8c6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34643 } } peers { permanent_uuid: "1d9aeb8d92ac439289b0193dee78ad72" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34291 } } peers { permanent_uuid: "55193fd102fc483ab492f9b5a184b225" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38129 } }
I20260924 21:10:59.070837 16351 ts_tablet_manager.cc:1428] T 4fe70b55ce09492691ac635e09462401 P 1d9aeb8d92ac439289b0193dee78ad72: Time spent starting tablet: real 0.002s user 0.004s sys 0.000s
Sep 24 21:10:59 dist-test-slave-c7tk krb5kdc[15694](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284259, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 21:10:59.077157 16462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:10:59.063363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:52295 (local address 127.0.98.62:39885)
0924 21:10:59.066172 (+ 2809us) server_negotiation.cc:206] Beginning negotiation
0924 21:10:59.066181 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 21:10:59.066216 (+ 35us) server_negotiation.cc:407] Connection header received
0924 21:10:59.066341 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:10:59.066347 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:10:59.066396 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:10:59.066534 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:10:59.067228 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:10:59.068457 (+ 1229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:10:59.070011 (+ 1554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:10:59.070517 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:10:59.070616 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:10:59.073221 (+ 2605us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 21:10:59.073224 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 21:10:59.073225 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 21:10:59.073242 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0924 21:10:59.075259 (+ 2017us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:10:59.075799 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:10:59.075801 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:10:59.075802 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:10:59.075903 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:10:59.076192 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:10:59.076192 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:10:59.076193 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:10:59.076446 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 21:10:59.076523 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0924 21:10:59.076889 (+ 366us) server_negotiation.cc:299] Negotiation successful
0924 21:10:59.077003 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2764,"thread_start_us":93,"threads_started":1}
I20260924 21:10:59.077207 16461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:10:59.062995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39885 (local address 127.0.98.1:52295)
0924 21:10:59.063320 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0924 21:10:59.063357 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0924 21:10:59.063852 (+ 495us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 21:10:59.066572 (+ 2720us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 21:10:59.066576 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 21:10:59.066885 (+ 309us) client_negotiation.cc:189] Negotiated authn=SASL
0924 21:10:59.067104 (+ 219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:10:59.067110 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:10:59.068590 (+ 1480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:10:59.068591 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:10:59.069923 (+ 1332us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:10:59.069926 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:10:59.070695 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:10:59.070696 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:10:59.070807 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:10:59.071365 (+ 558us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 21:10:59.071374 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 21:10:59.073102 (+ 1728us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 21:10:59.075375 (+ 2273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 21:10:59.075382 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 21:10:59.075384 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 21:10:59.075703 (+ 319us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 21:10:59.075978 (+ 275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 21:10:59.075979 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 21:10:59.075979 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 21:10:59.076117 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 21:10:59.076523 (+ 406us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 21:10:59.076529 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 21:10:59.076689 (+ 160us) client_negotiation.cc:769] Sending connection context
0924 21:10:59.076829 (+ 140us) client_negotiation.cc:240] Negotiation successful
0924 21:10:59.076990 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":290,"thread_start_us":88,"threads_started":1}
I20260924 21:10:59.078370 16459 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39885
I20260924 21:10:59.078449 16459 heartbeater.cc:464] Registering TS with master...
I20260924 21:10:59.078590 16459 heartbeater.cc:511] Master 127.0.98.62:39885 requested a full tablet report, sending...
I20260924 21:10:59.079170 16271 ts_manager.cc:194] Registered new tserver with Master: 1d9aeb8d92ac439289b0193dee78ad72 (127.0.98.1:34291)
I20260924 21:10:59.080087 16271 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:52295
I20260924 21:10:59.080822 16459 heartbeater.cc:503] Master 127.0.98.62:39885 was elected leader, sending a full tablet report...
Sep 24 21:10:59 dist-test-slave-c7tk krb5kdc[15694](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284258, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260924 21:10:59.087209 16466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:10:59.079648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34291 (local address 127.0.0.1:45318)
0924 21:10:59.079907 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0924 21:10:59.079936 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0924 21:10:59.079999 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 21:10:59.080216 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 21:10:59.080219 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 21:10:59.080489 (+ 270us) client_negotiation.cc:189] Negotiated authn=SASL
0924 21:10:59.080532 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:10:59.080534 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:10:59.081621 (+ 1087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:10:59.081621 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:10:59.082215 (+ 594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:10:59.082217 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:10:59.082707 (+ 490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:10:59.082709 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:10:59.082761 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:10:59.083121 (+ 360us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 21:10:59.083125 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 21:10:59.084412 (+ 1287us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 21:10:59.086216 (+ 1804us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 21:10:59.086221 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 21:10:59.086223 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 21:10:59.086419 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 21:10:59.086617 (+ 198us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 21:10:59.086618 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 21:10:59.086618 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 21:10:59.086668 (+ 50us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 21:10:59.086959 (+ 291us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 21:10:59.086964 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 21:10:59.087001 (+ 37us) client_negotiation.cc:769] Sending connection context
0924 21:10:59.087061 (+ 60us) client_negotiation.cc:240] Negotiation successful
0924 21:10:59.087138 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":75,"threads_started":1}
I20260924 21:10:59.087306 16467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:10:59.079720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45318 (local address 127.0.98.1:34291)
0924 21:10:59.079845 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0924 21:10:59.079849 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 21:10:59.080005 (+ 156us) server_negotiation.cc:407] Connection header received
0924 21:10:59.080062 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:10:59.080064 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:10:59.080076 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:10:59.080138 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:10:59.080601 (+ 463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:10:59.081549 (+ 948us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:10:59.082277 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:10:59.082644 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:10:59.082704 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:10:59.084497 (+ 1793us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 21:10:59.084505 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 21:10:59.084505 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 21:10:59.084512 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 21:10:59.086138 (+ 1626us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:10:59.086494 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:10:59.086496 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:10:59.086497 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:10:59.086554 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:10:59.086723 (+ 169us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:10:59.086724 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:10:59.086724 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:10:59.086898 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 21:10:59.086973 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0924 21:10:59.087105 (+ 132us) server_negotiation.cc:299] Negotiation successful
0924 21:10:59.087206 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105,"thread_start_us":71,"threads_started":1}
I20260924 21:10:59.087774 16393 tablet_service.cc:1514] Processing DeleteTablet for tablet 4fe70b55ce09492691ac635e09462401 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 21:10:38 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:45318
I20260924 21:10:59.088207 16468 tablet_replica.cc:330] stopping tablet replica
I20260924 21:10:59.088299 16468 raft_consensus.cc:2229] T 4fe70b55ce09492691ac635e09462401 P 1d9aeb8d92ac439289b0193dee78ad72 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 21:10:59.088366 16468 raft_consensus.cc:2258] T 4fe70b55ce09492691ac635e09462401 P 1d9aeb8d92ac439289b0193dee78ad72 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 21:10:59.088596 16468 ts_tablet_manager.cc:1905] T 4fe70b55ce09492691ac635e09462401 P 1d9aeb8d92ac439289b0193dee78ad72: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 21:10:59.090682 16468 ts_tablet_manager.cc:1918] T 4fe70b55ce09492691ac635e09462401 P 1d9aeb8d92ac439289b0193dee78ad72: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 21:10:59.090744 16468 log.cc:1196] T 4fe70b55ce09492691ac635e09462401 P 1d9aeb8d92ac439289b0193dee78ad72: Deleting WAL directory at /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790284076800039-392-0/minicluster-data/ts-0/wal/wals/4fe70b55ce09492691ac635e09462401
I20260924 21:10:59.090935 16468 ts_tablet_manager.cc:1939] T 4fe70b55ce09492691ac635e09462401 P 1d9aeb8d92ac439289b0193dee78ad72: Deleting consensus metadata
I20260924 21:10:59.091325 16256 catalog_manager.cc:4841] TS 1d9aeb8d92ac439289b0193dee78ad72 (127.0.98.1:34291): tablet 4fe70b55ce09492691ac635e09462401 (table test-table [id=8f52a32a3a8e4afda4b62b9791c2a98d]) successfully deleted
W20260924 21:10:59.147733 16464 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 21:10:59.147907 16464 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 21:10:59.147938 16464 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 21:10:59.148924 16464 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 21:10:59.148978 16464 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 21:10:59.149016 16464 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 21:10:59.150341 16464 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44143
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790284076800039-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790284076800039-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790284076800039-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:38129
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790284076800039-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=38699
--tserver_master_addrs=127.0.98.62:39885
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16464
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790284076800039-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 19:49:06 UTC on ccycloud.quasar-vhqjoc.root.comops.site
build id 4670864
I20260924 21:10:59.150588 16464 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 21:10:59.150782 16464 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 21:10:59.155087 16473 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 21:10:59.155113 16474 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 21:10:59.155108 16476 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 21:10:59.155386 16464 server_base.cc:1034] running on GCE node
I20260924 21:10:59.155571 16464 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 21:10:59.155824 16464 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 21:10:59.156970 16464 hybrid_clock.cc:648] HybridClock initialized: now 1790284259156960 us; error 24 us; skew 500 ppm
Sep 24 21:10:59 dist-test-slave-c7tk krb5kdc[15694](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284259, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 21:10:59.159045 16464 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 21:10:59.159440 16464 webserver.cc:460] Webserver started at http://127.0.98.2:38699/ using document root <none> and password file <none>
I20260924 21:10:59.159657 16464 fs_manager.cc:359] Metadata directory not provided
I20260924 21:10:59.159727 16464 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 21:10:59.161103 16464 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 21:10:59.161742 16483 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 21:10:59.161940 16464 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 21:10:59.162011 16464 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790284076800039-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790284076800039-392-0/minicluster-data/ts-1/wal
uuid: "55193fd102fc483ab492f9b5a184b225"
format_stamp: "Formatted at 2026-09-24 21:10:38 on dist-test-slave-c7tk"
server_key: "010dc19283e82651e052ef7c67606224"
server_key_iv: "d271153449d1e8c7437aa1571c4a91a5"
server_key_version: "encryptionkey@0"
I20260924 21:10:59.162262 16464 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790284076800039-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790284076800039-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790284076800039-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 21:10:59.178220 16464 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 21:10:59.179795 16464 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 21:10:59.179920 16464 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 21:10:59.180091 16464 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 21:10:59.180420 16490 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260924 21:10:59.181481 16464 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260924 21:10:59.181530 16464 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.001s user 0.000s sys 0.000s
I20260924 21:10:59.181564 16464 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260924 21:10:59.182148 16464 ts_tablet_manager.cc:610] Registered 1 tablets
I20260924 21:10:59.182204 16464 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260924 21:10:59.182242 16490 tablet_bootstrap.cc:492] T 4fe70b55ce09492691ac635e09462401 P 55193fd102fc483ab492f9b5a184b225: Bootstrap starting.
I20260924 21:10:59.188309 16464 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:38129
I20260924 21:10:59.188418 16597 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:38129 every 8 connection(s)
I20260924 21:10:59.188649 16464 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790284076800039-392-0/minicluster-data/ts-1/data/info.pb
I20260924 21:10:59.192361 16490 log.cc:824] T 4fe70b55ce09492691ac635e09462401 P 55193fd102fc483ab492f9b5a184b225: Log is configured to *not* fsync() on all Append() calls
I20260924 21:10:59.193890 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAzNNjS/build/release/bin/kudu as pid 16464
I20260924 21:10:59.194185 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAzNNjS/build/release/bin/kudu
/tmp/dist-test-taskAzNNjS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790284076800039-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790284076800039-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790284076800039-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790284076800039-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:34643
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:39885
--webserver_port=44675
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790284076800039-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44143
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790284076800039-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790284076800039-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790284076800039-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 21:10:59.194796 16490 tablet_bootstrap.cc:492] T 4fe70b55ce09492691ac635e09462401 P 55193fd102fc483ab492f9b5a184b225: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 21:10:59.195106 16490 tablet_bootstrap.cc:492] T 4fe70b55ce09492691ac635e09462401 P 55193fd102fc483ab492f9b5a184b225: Bootstrap complete.
Sep 24 21:10:59 dist-test-slave-c7tk krb5kdc[15694](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284259, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 21:10:59.196194 16490 ts_tablet_manager.cc:1397] T 4fe70b55ce09492691ac635e09462401 P 55193fd102fc483ab492f9b5a184b225: Time spent bootstrapping tablet: real 0.014s user 0.001s sys 0.010s
I20260924 21:10:59.196959 16490 raft_consensus.cc:348] T 4fe70b55ce09492691ac635e09462401 P 55193fd102fc483ab492f9b5a184b225 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e99c21581cf14320a2db35bd3ba7e8c6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34643 } } peers { permanent_uuid: "1d9aeb8d92ac439289b0193dee78ad72" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34291 } } peers { permanent_uuid: "55193fd102fc483ab492f9b5a184b225" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38129 } }
I20260924 21:10:59.197160 16490 raft_consensus.cc:729] T 4fe70b55ce09492691ac635e09462401 P 55193fd102fc483ab492f9b5a184b225 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 55193fd102fc483ab492f9b5a184b225, State: Initialized, Role: FOLLOWER
I20260924 21:10:59.197283 16490 consensus_queue.cc:260] T 4fe70b55ce09492691ac635e09462401 P 55193fd102fc483ab492f9b5a184b225 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e99c21581cf14320a2db35bd3ba7e8c6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34643 } } peers { permanent_uuid: "1d9aeb8d92ac439289b0193dee78ad72" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34291 } } peers { permanent_uuid: "55193fd102fc483ab492f9b5a184b225" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38129 } }
I20260924 21:10:59.197502 16490 ts_tablet_manager.cc:1428] T 4fe70b55ce09492691ac635e09462401 P 55193fd102fc483ab492f9b5a184b225: Time spent starting tablet: real 0.001s user 0.003s sys 0.000s
I20260924 21:10:59.198773 16462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:10:59.189322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:42599 (local address 127.0.98.62:39885)
0924 21:10:59.189383 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0924 21:10:59.189385 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 21:10:59.189550 (+ 165us) server_negotiation.cc:407] Connection header received
0924 21:10:59.190395 (+ 845us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:10:59.190396 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:10:59.190406 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:10:59.190455 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:10:59.191360 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:10:59.191856 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:10:59.193241 (+ 1385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:10:59.193436 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:10:59.193524 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:10:59.196087 (+ 2563us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 21:10:59.196089 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 21:10:59.196090 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 21:10:59.196098 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 21:10:59.197466 (+ 1368us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:10:59.197869 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:10:59.197870 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:10:59.197870 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:10:59.197927 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:10:59.198208 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:10:59.198209 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:10:59.198210 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:10:59.198313 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 21:10:59.198359 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0924 21:10:59.198647 (+ 288us) server_negotiation.cc:299] Negotiation successful
0924 21:10:59.198716 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260924 21:10:59.199004 16600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:10:59.189271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39885 (local address 127.0.98.2:42599)
0924 21:10:59.189430 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0924 21:10:59.189469 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0924 21:10:59.190289 (+ 820us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 21:10:59.190533 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 21:10:59.190539 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 21:10:59.190988 (+ 449us) client_negotiation.cc:189] Negotiated authn=SASL
0924 21:10:59.191276 (+ 288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:10:59.191282 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:10:59.191959 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:10:59.191960 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:10:59.193155 (+ 1195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:10:59.193157 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:10:59.193558 (+ 401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:10:59.193559 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:10:59.193681 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:10:59.194259 (+ 578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 21:10:59.194275 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 21:10:59.195996 (+ 1721us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 21:10:59.197517 (+ 1521us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 21:10:59.197520 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 21:10:59.197522 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 21:10:59.197786 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 21:10:59.198014 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 21:10:59.198014 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 21:10:59.198015 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 21:10:59.198140 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 21:10:59.198385 (+ 245us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 21:10:59.198390 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 21:10:59.198529 (+ 139us) client_negotiation.cc:769] Sending connection context
0924 21:10:59.198628 (+ 99us) client_negotiation.cc:240] Negotiation successful
0924 21:10:59.198791 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":93,"threads_started":1}
I20260924 21:10:59.199440 16598 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39885
I20260924 21:10:59.199530 16598 heartbeater.cc:464] Registering TS with master...
I20260924 21:10:59.199690 16598 heartbeater.cc:511] Master 127.0.98.62:39885 requested a full tablet report, sending...
I20260924 21:10:59.200184 16271 ts_manager.cc:194] Registered new tserver with Master: 55193fd102fc483ab492f9b5a184b225 (127.0.98.2:38129)
I20260924 21:10:59.200711 16271 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:42599
I20260924 21:10:59.201355 16598 heartbeater.cc:503] Master 127.0.98.62:39885 was elected leader, sending a full tablet report...
Sep 24 21:10:59 dist-test-slave-c7tk krb5kdc[15694](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284258, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.2@KRBTEST.COM
I20260924 21:10:59.207469 16604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:10:59.200654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60248 (local address 127.0.98.2:38129)
0924 21:10:59.200795 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0924 21:10:59.200798 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 21:10:59.200807 (+ 9us) server_negotiation.cc:407] Connection header received
0924 21:10:59.200837 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:10:59.200839 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:10:59.200854 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:10:59.200922 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:10:59.201223 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:10:59.202149 (+ 926us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:10:59.203062 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:10:59.203282 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:10:59.203325 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:10:59.205046 (+ 1721us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 21:10:59.205053 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 21:10:59.205053 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 21:10:59.205060 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 21:10:59.206447 (+ 1387us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:10:59.206772 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:10:59.206774 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:10:59.206774 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:10:59.206824 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:10:59.206996 (+ 172us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:10:59.206997 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:10:59.206998 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:10:59.207109 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 21:10:59.207185 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0924 21:10:59.207332 (+ 147us) server_negotiation.cc:299] Negotiation successful
0924 21:10:59.207401 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113,"thread_start_us":78,"threads_started":1}
I20260924 21:10:59.207466 16466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:10:59.200469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38129 (local address 127.0.0.1:60248)
0924 21:10:59.200507 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0924 21:10:59.200517 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 21:10:59.200558 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 21:10:59.200939 (+ 381us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 21:10:59.200940 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 21:10:59.201099 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0924 21:10:59.201143 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:10:59.201144 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:10:59.202220 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:10:59.202221 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:10:59.202985 (+ 764us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:10:59.202988 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:10:59.203344 (+ 356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:10:59.203347 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:10:59.203414 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:10:59.203707 (+ 293us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 21:10:59.203712 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 21:10:59.204952 (+ 1240us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 21:10:59.206538 (+ 1586us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 21:10:59.206539 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 21:10:59.206539 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 21:10:59.206687 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 21:10:59.206890 (+ 203us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 21:10:59.206891 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 21:10:59.206891 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 21:10:59.206939 (+ 48us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 21:10:59.207188 (+ 249us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 21:10:59.207189 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 21:10:59.207225 (+ 36us) client_negotiation.cc:769] Sending connection context
0924 21:10:59.207302 (+ 77us) client_negotiation.cc:240] Negotiation successful
0924 21:10:59.207390 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":20}
I20260924 21:10:59.207765 16532 tablet_service.cc:1514] Processing DeleteTablet for tablet 4fe70b55ce09492691ac635e09462401 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 21:10:38 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:60248
I20260924 21:10:59.208132 16605 tablet_replica.cc:330] stopping tablet replica
I20260924 21:10:59.208216 16605 raft_consensus.cc:2229] T 4fe70b55ce09492691ac635e09462401 P 55193fd102fc483ab492f9b5a184b225 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 21:10:59.208266 16605 raft_consensus.cc:2258] T 4fe70b55ce09492691ac635e09462401 P 55193fd102fc483ab492f9b5a184b225 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 21:10:59.208482 16605 ts_tablet_manager.cc:1905] T 4fe70b55ce09492691ac635e09462401 P 55193fd102fc483ab492f9b5a184b225: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 21:10:59.210700 16605 ts_tablet_manager.cc:1918] T 4fe70b55ce09492691ac635e09462401 P 55193fd102fc483ab492f9b5a184b225: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 21:10:59.210772 16605 log.cc:1196] T 4fe70b55ce09492691ac635e09462401 P 55193fd102fc483ab492f9b5a184b225: Deleting WAL directory at /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790284076800039-392-0/minicluster-data/ts-1/wal/wals/4fe70b55ce09492691ac635e09462401
I20260924 21:10:59.210964 16605 ts_tablet_manager.cc:1939] T 4fe70b55ce09492691ac635e09462401 P 55193fd102fc483ab492f9b5a184b225: Deleting consensus metadata
I20260924 21:10:59.211355 16256 catalog_manager.cc:4841] TS 55193fd102fc483ab492f9b5a184b225 (127.0.98.2:38129): tablet 4fe70b55ce09492691ac635e09462401 (table test-table [id=8f52a32a3a8e4afda4b62b9791c2a98d]) successfully deleted
W20260924 21:10:59.269526 16602 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 21:10:59.269677 16602 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 21:10:59.269696 16602 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 21:10:59.270689 16602 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 21:10:59.270728 16602 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 21:10:59.270751 16602 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 21:10:59.272073 16602 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44143
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790284076800039-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790284076800039-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790284076800039-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:34643
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790284076800039-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=44675
--tserver_master_addrs=127.0.98.62:39885
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16602
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790284076800039-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 19:49:06 UTC on ccycloud.quasar-vhqjoc.root.comops.site
build id 4670864
I20260924 21:10:59.272217 16602 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 21:10:59.272428 16602 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 21:10:59.276823 16610 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 21:10:59.276856 16611 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 21:10:59.276855 16613 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 21:10:59.277035 16602 server_base.cc:1034] running on GCE node
I20260924 21:10:59.277230 16602 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 21:10:59.277390 16602 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 21:10:59.278512 16602 hybrid_clock.cc:648] HybridClock initialized: now 1790284259278503 us; error 22 us; skew 500 ppm
Sep 24 21:10:59 dist-test-slave-c7tk krb5kdc[15694](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284259, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 21:10:59.280491 16602 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 21:10:59.280787 16602 webserver.cc:460] Webserver started at http://127.0.98.3:44675/ using document root <none> and password file <none>
I20260924 21:10:59.280936 16602 fs_manager.cc:359] Metadata directory not provided
I20260924 21:10:59.280978 16602 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 21:10:59.281989 16602 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 21:10:59.282406 16620 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 21:10:59.282546 16602 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 21:10:59.282624 16602 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790284076800039-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790284076800039-392-0/minicluster-data/ts-2/wal
uuid: "e99c21581cf14320a2db35bd3ba7e8c6"
format_stamp: "Formatted at 2026-09-24 21:10:38 on dist-test-slave-c7tk"
server_key: "98c6f865964e69523d7f33892db0edea"
server_key_iv: "f396dd3f02cffa8e0103119e0154437a"
server_key_version: "encryptionkey@0"
I20260924 21:10:59.282867 16602 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790284076800039-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790284076800039-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790284076800039-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 21:10:59.298475 16602 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 21:10:59.299831 16602 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 21:10:59.299957 16602 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 21:10:59.300123 16602 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 21:10:59.300351 16602 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 21:10:59.300396 16602 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 21:10:59.300429 16602 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 21:10:59.300451 16602 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 21:10:59.305419 16602 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:34643
I20260924 21:10:59.305509 16733 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:34643 every 8 connection(s)
I20260924 21:10:59.305908 16602 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790284076800039-392-0/minicluster-data/ts-2/data/info.pb
I20260924 21:10:59.310214 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAzNNjS/build/release/bin/kudu as pid 16602
Sep 24 21:10:59 dist-test-slave-c7tk krb5kdc[15694](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284259, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 21:10:59.314610 16462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:10:59.306521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:48975 (local address 127.0.98.62:39885)
0924 21:10:59.306580 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0924 21:10:59.306582 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 21:10:59.306732 (+ 150us) server_negotiation.cc:407] Connection header received
0924 21:10:59.307148 (+ 416us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:10:59.307149 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:10:59.307156 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:10:59.307204 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:10:59.307982 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:10:59.308384 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:10:59.309167 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:10:59.309358 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:10:59.309410 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:10:59.311422 (+ 2012us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 21:10:59.311424 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 21:10:59.311424 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 21:10:59.311431 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 21:10:59.312816 (+ 1385us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:10:59.313314 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:10:59.313315 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:10:59.313316 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:10:59.313408 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:10:59.313963 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:10:59.313964 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:10:59.313964 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:10:59.314104 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 21:10:59.314174 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0924 21:10:59.314420 (+ 246us) server_negotiation.cc:299] Negotiation successful
0924 21:10:59.314498 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260924 21:10:59.314625 16736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:10:59.306499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39885 (local address 127.0.98.3:48975)
0924 21:10:59.306625 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0924 21:10:59.306658 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0924 21:10:59.307041 (+ 383us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 21:10:59.307261 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 21:10:59.307264 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 21:10:59.307747 (+ 483us) client_negotiation.cc:189] Negotiated authn=SASL
0924 21:10:59.307909 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:10:59.307912 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:10:59.308446 (+ 534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:10:59.308446 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:10:59.309102 (+ 656us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:10:59.309103 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:10:59.309428 (+ 325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:10:59.309428 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:10:59.309511 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:10:59.309915 (+ 404us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 21:10:59.309922 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0924 21:10:59.311349 (+ 1427us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 21:10:59.312881 (+ 1532us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 21:10:59.312886 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 21:10:59.312887 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 21:10:59.313201 (+ 314us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 21:10:59.313481 (+ 280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 21:10:59.313481 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 21:10:59.313482 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 21:10:59.313595 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 21:10:59.314180 (+ 585us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 21:10:59.314184 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 21:10:59.314290 (+ 106us) client_negotiation.cc:769] Sending connection context
0924 21:10:59.314365 (+ 75us) client_negotiation.cc:240] Negotiation successful
0924 21:10:59.314467 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":93,"thread_start_us":71,"threads_started":1}
I20260924 21:10:59.315255 16734 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39885
I20260924 21:10:59.315356 16734 heartbeater.cc:464] Registering TS with master...
I20260924 21:10:59.315516 16734 heartbeater.cc:511] Master 127.0.98.62:39885 requested a full tablet report, sending...
I20260924 21:10:59.315845 16271 ts_manager.cc:194] Registered new tserver with Master: e99c21581cf14320a2db35bd3ba7e8c6 (127.0.98.3:34643)
I20260924 21:10:59.316186 16271 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:48975
I20260924 21:10:59.317454 16737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:10:59.310572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45350 (local address 127.0.98.62:39885)
0924 21:10:59.310677 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0924 21:10:59.310679 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 21:10:59.310798 (+ 119us) server_negotiation.cc:407] Connection header received
0924 21:10:59.310846 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:10:59.310847 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:10:59.310858 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:10:59.310917 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:10:59.311491 (+ 574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:10:59.312110 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:10:59.313113 (+ 1003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:10:59.313394 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:10:59.313469 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:10:59.314902 (+ 1433us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 21:10:59.314902 (+ 0us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 21:10:59.314903 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 21:10:59.314909 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 21:10:59.316465 (+ 1556us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:10:59.316767 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:10:59.316768 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:10:59.316768 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:10:59.316817 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:10:59.316993 (+ 176us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:10:59.316993 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:10:59.316994 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:10:59.317085 (+ 91us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 21:10:59.317266 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0924 21:10:59.317298 (+ 32us) server_negotiation.cc:299] Negotiation successful
0924 21:10:59.317368 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88,"thread_start_us":53,"threads_started":1}
I20260924 21:10:59.318148 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 21:10:59.320147 16737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:10:59.318453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45362 (local address 127.0.98.62:39885)
0924 21:10:59.318484 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0924 21:10:59.318486 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 21:10:59.318555 (+ 69us) server_negotiation.cc:407] Connection header received
0924 21:10:59.318613 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:10:59.318614 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:10:59.318634 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:10:59.318689 (+ 55us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 21:10:59.318757 (+ 68us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:10:59.319149 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:10:59.319703 (+ 554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:10:59.319855 (+ 152us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:10:59.319896 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:10:59.319992 (+ 96us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 21:10:59.320021 (+ 29us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0924 21:10:59.320115 (+ 94us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:45362: token expired
Metrics: {"server-negotiator.queue_time_us":16}
W20260924 21:10:59.320202 16737 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:45362: token expired
W20260924 21:10:59.320264 16203 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.0.98.62:39885)
I20260924 21:10:59.324753 16737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:10:59.320464 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45370 (local address 127.0.98.62:39885)
0924 21:10:59.320490 (+ 26us) server_negotiation.cc:206] Beginning negotiation
0924 21:10:59.320492 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 21:10:59.320501 (+ 9us) server_negotiation.cc:407] Connection header received
0924 21:10:59.320516 (+ 15us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:10:59.320516 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:10:59.320522 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:10:59.320572 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:10:59.320906 (+ 334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:10:59.321282 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:10:59.321885 (+ 603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:10:59.322037 (+ 152us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:10:59.322080 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:10:59.323005 (+ 925us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 21:10:59.323005 (+ 0us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 21:10:59.323005 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 21:10:59.323009 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 21:10:59.324020 (+ 1011us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:10:59.324311 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:10:59.324312 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:10:59.324312 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:10:59.324353 (+ 41us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:10:59.324461 (+ 108us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:10:59.324461 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:10:59.324461 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:10:59.324533 (+ 72us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 21:10:59.324592 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0924 21:10:59.324650 (+ 58us) server_negotiation.cc:299] Negotiation successful
0924 21:10:59.324699 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12}
I20260924 21:10:59.325745 16271 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45370:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 21:10:59.327103 16271 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 21:10:59.328873 16393 tablet_service.cc:1467] Processing CreateTablet for tablet 33e360f293484359a0843fa409a630e3 (DEFAULT_TABLE table=test-table [id=960ee76a8b6b446188b1b1c5beb2ce46]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 21:10:59.328930 16532 tablet_service.cc:1467] Processing CreateTablet for tablet 33e360f293484359a0843fa409a630e3 (DEFAULT_TABLE table=test-table [id=960ee76a8b6b446188b1b1c5beb2ce46]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 21:10:59.329048 16393 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 33e360f293484359a0843fa409a630e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 21:10:59.329104 16532 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 33e360f293484359a0843fa409a630e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 21:10:59.330430 16351 tablet_bootstrap.cc:492] T 33e360f293484359a0843fa409a630e3 P 1d9aeb8d92ac439289b0193dee78ad72: Bootstrap starting.
I20260924 21:10:59.330636 16490 tablet_bootstrap.cc:492] T 33e360f293484359a0843fa409a630e3 P 55193fd102fc483ab492f9b5a184b225: Bootstrap starting.
I20260924 21:10:59.330775 16351 tablet_bootstrap.cc:654] T 33e360f293484359a0843fa409a630e3 P 1d9aeb8d92ac439289b0193dee78ad72: Neither blocks nor log segments found. Creating new log.
I20260924 21:10:59.330919 16490 tablet_bootstrap.cc:654] T 33e360f293484359a0843fa409a630e3 P 55193fd102fc483ab492f9b5a184b225: Neither blocks nor log segments found. Creating new log.
I20260924 21:10:59.331341 16351 tablet_bootstrap.cc:492] T 33e360f293484359a0843fa409a630e3 P 1d9aeb8d92ac439289b0193dee78ad72: No bootstrap required, opened a new log
I20260924 21:10:59.331393 16351 ts_tablet_manager.cc:1397] T 33e360f293484359a0843fa409a630e3 P 1d9aeb8d92ac439289b0193dee78ad72: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260924 21:10:59.331717 16351 raft_consensus.cc:348] T 33e360f293484359a0843fa409a630e3 P 1d9aeb8d92ac439289b0193dee78ad72 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e99c21581cf14320a2db35bd3ba7e8c6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34643 } } peers { permanent_uuid: "1d9aeb8d92ac439289b0193dee78ad72" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34291 } } peers { permanent_uuid: "55193fd102fc483ab492f9b5a184b225" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38129 } }
I20260924 21:10:59.331748 16490 tablet_bootstrap.cc:492] T 33e360f293484359a0843fa409a630e3 P 55193fd102fc483ab492f9b5a184b225: No bootstrap required, opened a new log
I20260924 21:10:59.331794 16490 ts_tablet_manager.cc:1397] T 33e360f293484359a0843fa409a630e3 P 55193fd102fc483ab492f9b5a184b225: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260924 21:10:59.331810 16351 raft_consensus.cc:374] T 33e360f293484359a0843fa409a630e3 P 1d9aeb8d92ac439289b0193dee78ad72 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 21:10:59.331875 16351 raft_consensus.cc:729] T 33e360f293484359a0843fa409a630e3 P 1d9aeb8d92ac439289b0193dee78ad72 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d9aeb8d92ac439289b0193dee78ad72, State: Initialized, Role: FOLLOWER
I20260924 21:10:59.331919 16351 consensus_queue.cc:260] T 33e360f293484359a0843fa409a630e3 P 1d9aeb8d92ac439289b0193dee78ad72 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e99c21581cf14320a2db35bd3ba7e8c6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34643 } } peers { permanent_uuid: "1d9aeb8d92ac439289b0193dee78ad72" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34291 } } peers { permanent_uuid: "55193fd102fc483ab492f9b5a184b225" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38129 } }
I20260924 21:10:59.331935 16490 raft_consensus.cc:348] T 33e360f293484359a0843fa409a630e3 P 55193fd102fc483ab492f9b5a184b225 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e99c21581cf14320a2db35bd3ba7e8c6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34643 } } peers { permanent_uuid: "1d9aeb8d92ac439289b0193dee78ad72" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34291 } } peers { permanent_uuid: "55193fd102fc483ab492f9b5a184b225" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38129 } }
I20260924 21:10:59.332016 16351 ts_tablet_manager.cc:1428] T 33e360f293484359a0843fa409a630e3 P 1d9aeb8d92ac439289b0193dee78ad72: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260924 21:10:59.332013 16490 raft_consensus.cc:374] T 33e360f293484359a0843fa409a630e3 P 55193fd102fc483ab492f9b5a184b225 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 21:10:59.332050 16490 raft_consensus.cc:729] T 33e360f293484359a0843fa409a630e3 P 55193fd102fc483ab492f9b5a184b225 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 55193fd102fc483ab492f9b5a184b225, State: Initialized, Role: FOLLOWER
I20260924 21:10:59.332109 16490 consensus_queue.cc:260] T 33e360f293484359a0843fa409a630e3 P 55193fd102fc483ab492f9b5a184b225 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e99c21581cf14320a2db35bd3ba7e8c6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34643 } } peers { permanent_uuid: "1d9aeb8d92ac439289b0193dee78ad72" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34291 } } peers { permanent_uuid: "55193fd102fc483ab492f9b5a184b225" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38129 } }
I20260924 21:10:59.332191 16490 ts_tablet_manager.cc:1428] T 33e360f293484359a0843fa409a630e3 P 55193fd102fc483ab492f9b5a184b225: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 21:10:59.332368 16466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:10:59.328603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:34643 (local address 127.0.0.1:51498)
0924 21:10:59.328644 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0924 21:10:59.328653 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 21:10:59.328687 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 21:10:59.328971 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 21:10:59.328972 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 21:10:59.328973 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 21:10:59.329007 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:10:59.329008 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:10:59.330165 (+ 1157us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:10:59.330166 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:10:59.331186 (+ 1020us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:10:59.331189 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:10:59.332101 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:10:59.332102 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:10:59.332167 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:10:59.332188 (+ 21us) client_negotiation.cc:769] Sending connection context
0924 21:10:59.332256 (+ 68us) client_negotiation.cc:240] Negotiation successful
0924 21:10:59.332295 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28}
I20260924 21:10:59.332499 16740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:10:59.328671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51498 (local address 127.0.98.3:34643)
0924 21:10:59.328823 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0924 21:10:59.328827 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 21:10:59.328835 (+ 8us) server_negotiation.cc:407] Connection header received
0924 21:10:59.328863 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:10:59.328865 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:10:59.328877 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:10:59.328978 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 21:10:59.329069 (+ 91us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:10:59.330057 (+ 988us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:10:59.331277 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:10:59.331930 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:10:59.332004 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:10:59.332059 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0924 21:10:59.332389 (+ 330us) server_negotiation.cc:299] Negotiation successful
0924 21:10:59.332427 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":90,"threads_started":1}
I20260924 21:10:59.333068 16668 tablet_service.cc:1467] Processing CreateTablet for tablet 33e360f293484359a0843fa409a630e3 (DEFAULT_TABLE table=test-table [id=960ee76a8b6b446188b1b1c5beb2ce46]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 21:10:59.333338 16668 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 33e360f293484359a0843fa409a630e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 21:10:59.335744 16743 tablet_bootstrap.cc:492] T 33e360f293484359a0843fa409a630e3 P e99c21581cf14320a2db35bd3ba7e8c6: Bootstrap starting.
I20260924 21:10:59.336216 16743 tablet_bootstrap.cc:654] T 33e360f293484359a0843fa409a630e3 P e99c21581cf14320a2db35bd3ba7e8c6: Neither blocks nor log segments found. Creating new log.
I20260924 21:10:59.336607 16743 log.cc:824] T 33e360f293484359a0843fa409a630e3 P e99c21581cf14320a2db35bd3ba7e8c6: Log is configured to *not* fsync() on all Append() calls
I20260924 21:10:59.337134 16743 tablet_bootstrap.cc:492] T 33e360f293484359a0843fa409a630e3 P e99c21581cf14320a2db35bd3ba7e8c6: No bootstrap required, opened a new log
I20260924 21:10:59.337181 16743 ts_tablet_manager.cc:1397] T 33e360f293484359a0843fa409a630e3 P e99c21581cf14320a2db35bd3ba7e8c6: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260924 21:10:59.338544 16743 raft_consensus.cc:348] T 33e360f293484359a0843fa409a630e3 P e99c21581cf14320a2db35bd3ba7e8c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e99c21581cf14320a2db35bd3ba7e8c6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34643 } } peers { permanent_uuid: "1d9aeb8d92ac439289b0193dee78ad72" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34291 } } peers { permanent_uuid: "55193fd102fc483ab492f9b5a184b225" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38129 } }
I20260924 21:10:59.338661 16743 raft_consensus.cc:374] T 33e360f293484359a0843fa409a630e3 P e99c21581cf14320a2db35bd3ba7e8c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 21:10:59.338681 16743 raft_consensus.cc:729] T 33e360f293484359a0843fa409a630e3 P e99c21581cf14320a2db35bd3ba7e8c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e99c21581cf14320a2db35bd3ba7e8c6, State: Initialized, Role: FOLLOWER
I20260924 21:10:59.338781 16743 consensus_queue.cc:260] T 33e360f293484359a0843fa409a630e3 P e99c21581cf14320a2db35bd3ba7e8c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e99c21581cf14320a2db35bd3ba7e8c6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34643 } } peers { permanent_uuid: "1d9aeb8d92ac439289b0193dee78ad72" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34291 } } peers { permanent_uuid: "55193fd102fc483ab492f9b5a184b225" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38129 } }
I20260924 21:10:59.338986 16743 ts_tablet_manager.cc:1428] T 33e360f293484359a0843fa409a630e3 P e99c21581cf14320a2db35bd3ba7e8c6: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 21:10:59.339025 16734 heartbeater.cc:503] Master 127.0.98.62:39885 was elected leader, sending a full tablet report...
I20260924 21:10:59.356608 16465 raft_consensus.cc:482] T 33e360f293484359a0843fa409a630e3 P 1d9aeb8d92ac439289b0193dee78ad72 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 21:10:59.356714 16465 raft_consensus.cc:504] T 33e360f293484359a0843fa409a630e3 P 1d9aeb8d92ac439289b0193dee78ad72 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e99c21581cf14320a2db35bd3ba7e8c6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34643 } } peers { permanent_uuid: "1d9aeb8d92ac439289b0193dee78ad72" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34291 } } peers { permanent_uuid: "55193fd102fc483ab492f9b5a184b225" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38129 } }
I20260924 21:10:59.357035 16465 leader_election.cc:288] T 33e360f293484359a0843fa409a630e3 P 1d9aeb8d92ac439289b0193dee78ad72 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e99c21581cf14320a2db35bd3ba7e8c6 (127.0.98.3:34643), 55193fd102fc483ab492f9b5a184b225 (127.0.98.2:38129)
I20260924 21:10:59.360140 16461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:10:59.357172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:34643 (local address 127.0.98.1:56983)
0924 21:10:59.357225 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0924 21:10:59.357255 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0924 21:10:59.357300 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 21:10:59.357474 (+ 174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 21:10:59.357475 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 21:10:59.357476 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 21:10:59.357521 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:10:59.357523 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:10:59.358273 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:10:59.358274 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:10:59.359264 (+ 990us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:10:59.359267 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:10:59.359935 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:10:59.359936 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:10:59.359983 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:10:59.359996 (+ 13us) client_negotiation.cc:769] Sending connection context
0924 21:10:59.360036 (+ 40us) client_negotiation.cc:240] Negotiation successful
0924 21:10:59.360066 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34}
I20260924 21:10:59.360163 16740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:10:59.357223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:56983 (local address 127.0.98.3:34643)
0924 21:10:59.357285 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0924 21:10:59.357287 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 21:10:59.357296 (+ 9us) server_negotiation.cc:407] Connection header received
0924 21:10:59.357404 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:10:59.357405 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:10:59.357414 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:10:59.357458 (+ 44us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 21:10:59.357580 (+ 122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:10:59.358214 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:10:59.359383 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:10:59.359873 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:10:59.359936 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:10:59.359943 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0924 21:10:59.360061 (+ 118us) server_negotiation.cc:299] Negotiation successful
0924 21:10:59.360095 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260924 21:10:59.360590 16746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:10:59.357315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38129 (local address 127.0.98.1:41453)
0924 21:10:59.357438 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0924 21:10:59.357447 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 21:10:59.357495 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 21:10:59.357634 (+ 139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 21:10:59.357638 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 21:10:59.357639 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 21:10:59.357687 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:10:59.357688 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:10:59.358433 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:10:59.358434 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:10:59.359470 (+ 1036us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:10:59.359472 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:10:59.360163 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:10:59.360164 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:10:59.360367 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:10:59.360371 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 21:10:59.360477 (+ 106us) client_negotiation.cc:240] Negotiation successful
0924 21:10:59.360519 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":82,"threads_started":1}
I20260924 21:10:59.360597 16604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:10:59.357360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41453 (local address 127.0.98.2:38129)
0924 21:10:59.357398 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0924 21:10:59.357400 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 21:10:59.357506 (+ 106us) server_negotiation.cc:407] Connection header received
0924 21:10:59.357533 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:10:59.357534 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:10:59.357562 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:10:59.357620 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 21:10:59.357742 (+ 122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:10:59.358377 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:10:59.359536 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:10:59.360078 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:10:59.360136 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:10:59.360203 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0924 21:10:59.360486 (+ 283us) server_negotiation.cc:299] Negotiation successful
0924 21:10:59.360529 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 21:10:59.360597 16688 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33e360f293484359a0843fa409a630e3" candidate_uuid: "1d9aeb8d92ac439289b0193dee78ad72" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e99c21581cf14320a2db35bd3ba7e8c6" is_pre_election: true
I20260924 21:10:59.360703 16688 raft_consensus.cc:2454] T 33e360f293484359a0843fa409a630e3 P e99c21581cf14320a2db35bd3ba7e8c6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d9aeb8d92ac439289b0193dee78ad72 in term 0.
I20260924 21:10:59.360867 16346 leader_election.cc:302] T 33e360f293484359a0843fa409a630e3 P 1d9aeb8d92ac439289b0193dee78ad72 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d9aeb8d92ac439289b0193dee78ad72, e99c21581cf14320a2db35bd3ba7e8c6; no voters:
I20260924 21:10:59.360853 16552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33e360f293484359a0843fa409a630e3" candidate_uuid: "1d9aeb8d92ac439289b0193dee78ad72" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "55193fd102fc483ab492f9b5a184b225" is_pre_election: true
I20260924 21:10:59.360968 16552 raft_consensus.cc:2454] T 33e360f293484359a0843fa409a630e3 P 55193fd102fc483ab492f9b5a184b225 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d9aeb8d92ac439289b0193dee78ad72 in term 0.
I20260924 21:10:59.360965 16465 raft_consensus.cc:2781] T 33e360f293484359a0843fa409a630e3 P 1d9aeb8d92ac439289b0193dee78ad72 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 21:10:59.361003 16465 raft_consensus.cc:482] T 33e360f293484359a0843fa409a630e3 P 1d9aeb8d92ac439289b0193dee78ad72 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 21:10:59.361037 16465 raft_consensus.cc:3037] T 33e360f293484359a0843fa409a630e3 P 1d9aeb8d92ac439289b0193dee78ad72 [term 0 FOLLOWER]: Advancing to term 1
I20260924 21:10:59.361567 16465 raft_consensus.cc:504] T 33e360f293484359a0843fa409a630e3 P 1d9aeb8d92ac439289b0193dee78ad72 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e99c21581cf14320a2db35bd3ba7e8c6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34643 } } peers { permanent_uuid: "1d9aeb8d92ac439289b0193dee78ad72" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34291 } } peers { permanent_uuid: "55193fd102fc483ab492f9b5a184b225" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38129 } }
I20260924 21:10:59.361675 16465 leader_election.cc:288] T 33e360f293484359a0843fa409a630e3 P 1d9aeb8d92ac439289b0193dee78ad72 [CANDIDATE]: Term 1 election: Requested vote from peers e99c21581cf14320a2db35bd3ba7e8c6 (127.0.98.3:34643), 55193fd102fc483ab492f9b5a184b225 (127.0.98.2:38129)
I20260924 21:10:59.361858 16552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33e360f293484359a0843fa409a630e3" candidate_uuid: "1d9aeb8d92ac439289b0193dee78ad72" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "55193fd102fc483ab492f9b5a184b225"
I20260924 21:10:59.361867 16688 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33e360f293484359a0843fa409a630e3" candidate_uuid: "1d9aeb8d92ac439289b0193dee78ad72" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e99c21581cf14320a2db35bd3ba7e8c6"
I20260924 21:10:59.361923 16688 raft_consensus.cc:3037] T 33e360f293484359a0843fa409a630e3 P e99c21581cf14320a2db35bd3ba7e8c6 [term 0 FOLLOWER]: Advancing to term 1
I20260924 21:10:59.361939 16552 raft_consensus.cc:3037] T 33e360f293484359a0843fa409a630e3 P 55193fd102fc483ab492f9b5a184b225 [term 0 FOLLOWER]: Advancing to term 1
I20260924 21:10:59.362474 16688 raft_consensus.cc:2454] T 33e360f293484359a0843fa409a630e3 P e99c21581cf14320a2db35bd3ba7e8c6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d9aeb8d92ac439289b0193dee78ad72 in term 1.
I20260924 21:10:59.362543 16552 raft_consensus.cc:2454] T 33e360f293484359a0843fa409a630e3 P 55193fd102fc483ab492f9b5a184b225 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d9aeb8d92ac439289b0193dee78ad72 in term 1.
I20260924 21:10:59.362615 16346 leader_election.cc:302] T 33e360f293484359a0843fa409a630e3 P 1d9aeb8d92ac439289b0193dee78ad72 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d9aeb8d92ac439289b0193dee78ad72, e99c21581cf14320a2db35bd3ba7e8c6; no voters:
I20260924 21:10:59.362706 16465 raft_consensus.cc:2781] T 33e360f293484359a0843fa409a630e3 P 1d9aeb8d92ac439289b0193dee78ad72 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 21:10:59.362846 16465 raft_consensus.cc:686] T 33e360f293484359a0843fa409a630e3 P 1d9aeb8d92ac439289b0193dee78ad72 [term 1 LEADER]: Becoming Leader. State: Replica: 1d9aeb8d92ac439289b0193dee78ad72, State: Running, Role: LEADER
I20260924 21:10:59.362900 16465 consensus_queue.cc:237] T 33e360f293484359a0843fa409a630e3 P 1d9aeb8d92ac439289b0193dee78ad72 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e99c21581cf14320a2db35bd3ba7e8c6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34643 } } peers { permanent_uuid: "1d9aeb8d92ac439289b0193dee78ad72" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34291 } } peers { permanent_uuid: "55193fd102fc483ab492f9b5a184b225" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38129 } }
I20260924 21:10:59.363467 16270 catalog_manager.cc:5495] T 33e360f293484359a0843fa409a630e3 P 1d9aeb8d92ac439289b0193dee78ad72 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1d9aeb8d92ac439289b0193dee78ad72 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "1d9aeb8d92ac439289b0193dee78ad72" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e99c21581cf14320a2db35bd3ba7e8c6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34643 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1d9aeb8d92ac439289b0193dee78ad72" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34291 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "55193fd102fc483ab492f9b5a184b225" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38129 } health_report { overall_health: UNKNOWN } } }
I20260924 21:10:59.367842 16467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:10:59.366186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45332 (local address 127.0.98.1:34291)
0924 21:10:59.366243 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0924 21:10:59.366244 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 21:10:59.366251 (+ 7us) server_negotiation.cc:407] Connection header received
0924 21:10:59.366298 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:10:59.366299 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:10:59.366310 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:10:59.366347 (+ 37us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 21:10:59.366414 (+ 67us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:10:59.366796 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:10:59.367349 (+ 553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:10:59.367501 (+ 152us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:10:59.367540 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:10:59.367641 (+ 101us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 21:10:59.367705 (+ 64us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 21:10:59.367749 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0924 21:10:59.367762 (+ 13us) server_negotiation.cc:299] Negotiation successful
0924 21:10:59.367775 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260924 21:10:59.368912 16688 raft_consensus.cc:1261] T 33e360f293484359a0843fa409a630e3 P e99c21581cf14320a2db35bd3ba7e8c6 [term 1 FOLLOWER]: Refusing update from remote peer 1d9aeb8d92ac439289b0193dee78ad72: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 21:10:59.368912 16552 raft_consensus.cc:1261] T 33e360f293484359a0843fa409a630e3 P 55193fd102fc483ab492f9b5a184b225 [term 1 FOLLOWER]: Refusing update from remote peer 1d9aeb8d92ac439289b0193dee78ad72: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 21:10:59.369167 16747 consensus_queue.cc:1035] T 33e360f293484359a0843fa409a630e3 P 1d9aeb8d92ac439289b0193dee78ad72 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e99c21581cf14320a2db35bd3ba7e8c6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 21:10:59.369306 16465 consensus_queue.cc:1035] T 33e360f293484359a0843fa409a630e3 P 1d9aeb8d92ac439289b0193dee78ad72 [LEADER]: Connected to new peer: Peer: permanent_uuid: "55193fd102fc483ab492f9b5a184b225" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38129 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 21:10:59.371304 16752 mvcc.cc:205] Tried to move back new op lower bound from 7333004326373277696 to 7333004326350929920. Current Snapshot: MvccSnapshot[applied={T|T < 7333004326373277696}]
I20260924 21:10:59.371623 16751 mvcc.cc:205] Tried to move back new op lower bound from 7333004326373277696 to 7333004326350929920. Current Snapshot: MvccSnapshot[applied={T|T < 7333004326373277696}]
I20260924 21:10:59.372279 16270 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45370:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 21:10:59.372350 16270 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45370:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 21:10:59.372877 16270 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a18533bad310410ca2873fde2d8ff491: Sending DeleteTablet for 3 replicas of tablet 33e360f293484359a0843fa409a630e3
I20260924 21:10:59.373227 16668 tablet_service.cc:1514] Processing DeleteTablet for tablet 33e360f293484359a0843fa409a630e3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 21:10:59 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:51498
I20260924 21:10:59.373288 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAzNNjS/build/release/bin/kudu with pid 16309
I20260924 21:10:59.373301 16532 tablet_service.cc:1514] Processing DeleteTablet for tablet 33e360f293484359a0843fa409a630e3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 21:10:59 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:60248
I20260924 21:10:59.373370 16605 tablet_replica.cc:330] T 33e360f293484359a0843fa409a630e3 P 55193fd102fc483ab492f9b5a184b225: stopping tablet replica
I20260924 21:10:59.373414 16605 raft_consensus.cc:2229] T 33e360f293484359a0843fa409a630e3 P 55193fd102fc483ab492f9b5a184b225 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 21:10:59.373466 16605 raft_consensus.cc:2258] T 33e360f293484359a0843fa409a630e3 P 55193fd102fc483ab492f9b5a184b225 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 21:10:59.374624 16759 tablet_replica.cc:330] T 33e360f293484359a0843fa409a630e3 P e99c21581cf14320a2db35bd3ba7e8c6: stopping tablet replica
I20260924 21:10:59.374734 16759 raft_consensus.cc:2229] T 33e360f293484359a0843fa409a630e3 P e99c21581cf14320a2db35bd3ba7e8c6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 21:10:59.374830 16759 raft_consensus.cc:2258] T 33e360f293484359a0843fa409a630e3 P e99c21581cf14320a2db35bd3ba7e8c6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 21:10:59.375180 16759 ts_tablet_manager.cc:1905] T 33e360f293484359a0843fa409a630e3 P e99c21581cf14320a2db35bd3ba7e8c6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 21:10:59.375816 16605 ts_tablet_manager.cc:1905] T 33e360f293484359a0843fa409a630e3 P 55193fd102fc483ab492f9b5a184b225: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 21:10:59.376751 16759 ts_tablet_manager.cc:1918] T 33e360f293484359a0843fa409a630e3 P e99c21581cf14320a2db35bd3ba7e8c6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 21:10:59.376849 16759 log.cc:1196] T 33e360f293484359a0843fa409a630e3 P e99c21581cf14320a2db35bd3ba7e8c6: Deleting WAL directory at /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790284076800039-392-0/minicluster-data/ts-2/wal/wals/33e360f293484359a0843fa409a630e3
I20260924 21:10:59.377116 16759 ts_tablet_manager.cc:1939] T 33e360f293484359a0843fa409a630e3 P e99c21581cf14320a2db35bd3ba7e8c6: Deleting consensus metadata
I20260924 21:10:59.377295 16605 ts_tablet_manager.cc:1918] T 33e360f293484359a0843fa409a630e3 P 55193fd102fc483ab492f9b5a184b225: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 21:10:59.377348 16605 log.cc:1196] T 33e360f293484359a0843fa409a630e3 P 55193fd102fc483ab492f9b5a184b225: Deleting WAL directory at /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790284076800039-392-0/minicluster-data/ts-1/wal/wals/33e360f293484359a0843fa409a630e3
I20260924 21:10:59.377452 16256 catalog_manager.cc:4841] TS e99c21581cf14320a2db35bd3ba7e8c6 (127.0.98.3:34643): tablet 33e360f293484359a0843fa409a630e3 (table test-table [id=960ee76a8b6b446188b1b1c5beb2ce46]) successfully deleted
I20260924 21:10:59.377524 16605 ts_tablet_manager.cc:1939] T 33e360f293484359a0843fa409a630e3 P 55193fd102fc483ab492f9b5a184b225: Deleting consensus metadata
I20260924 21:10:59.378006 16256 catalog_manager.cc:4841] TS 55193fd102fc483ab492f9b5a184b225 (127.0.98.2:38129): tablet 33e360f293484359a0843fa409a630e3 (table test-table [id=960ee76a8b6b446188b1b1c5beb2ce46]) successfully deleted
W20260924 21:10:59.379158 16256 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:34291 (error 108)
I20260924 21:10:59.379379 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAzNNjS/build/release/bin/kudu with pid 16464
I20260924 21:10:59.379865 16466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:10:59.379505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34291 (local address 127.0.0.1:45340)
0924 21:10:59.379802 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0924 21:10:59.379836 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:34291: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":282}
W20260924 21:10:59.380010 16256 catalog_manager.cc:4584] TS 1d9aeb8d92ac439289b0193dee78ad72 (127.0.98.1:34291): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 33e360f293484359a0843fa409a630e3: Network error: Client connection negotiation failed: client connection to 127.0.98.1:34291: connect: Connection refused (error 111)
I20260924 21:10:59.384153 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAzNNjS/build/release/bin/kudu with pid 16602
I20260924 21:10:59.388008 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAzNNjS/build/release/bin/kudu with pid 16238
I20260924 21:11:19.393078 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAzNNjS/build/release/bin/kudu
/tmp/dist-test-taskAzNNjS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790284076800039-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790284076800039-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790284076800039-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790284076800039-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39885
--webserver_interface=127.0.98.62
--webserver_port=38395
--keytab_file=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790284076800039-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44143
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39885
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790284076800039-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790284076800039-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790284076800039-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 21:11:19.464197 16761 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 21:11:19.464350 16761 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 21:11:19.464368 16761 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 21:11:19.465255 16761 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260924 21:11:19.465282 16761 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260924 21:11:19.465293 16761 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260924 21:11:19.465303 16761 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 21:11:19.465313 16761 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 21:11:19.465325 16761 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 21:11:19.465337 16761 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 21:11:19.465351 16761 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 21:11:19.466665 16761 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44143
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790284076800039-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790284076800039-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39885
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790284076800039-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39885
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790284076800039-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=38395
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16761
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790284076800039-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 19:49:06 UTC on ccycloud.quasar-vhqjoc.root.comops.site
build id 4670864
I20260924 21:11:19.466881 16761 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 21:11:19.467099 16761 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 21:11:19.471570 16761 server_base.cc:1034] running on GCE node
W20260924 21:11:19.471679 16767 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 21:11:19.471679 16769 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 21:11:19.471767 16766 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 21:11:19.472009 16761 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 21:11:19.472249 16761 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 21:11:19.473382 16761 hybrid_clock.cc:648] HybridClock initialized: now 1790284279473373 us; error 25 us; skew 500 ppm
Sep 24 21:11:19 dist-test-slave-c7tk krb5kdc[15694](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284279, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 21:11:19.479661 16761 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 21:11:19.480000 16761 webserver.cc:460] Webserver started at http://127.0.98.62:38395/ using document root <none> and password file <none>
I20260924 21:11:19.480178 16761 fs_manager.cc:359] Metadata directory not provided
I20260924 21:11:19.480216 16761 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 21:11:19.481348 16761 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 21:11:19.481911 16776 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 21:11:19.482048 16761 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 21:11:19.482143 16761 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790284076800039-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790284076800039-392-0/minicluster-data/master-0/wal
uuid: "a18533bad310410ca2873fde2d8ff491"
format_stamp: "Formatted at 2026-09-24 21:10:38 on dist-test-slave-c7tk"
server_key: "7a801cb9420e22bcf66f976e7e25d2fd"
server_key_iv: "db9a9a6f8c42619d34d226e73332aed5"
server_key_version: "encryptionkey@0"
I20260924 21:11:19.482393 16761 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790284076800039-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790284076800039-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790284076800039-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 21:11:19.499512 16761 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 21:11:19.500878 16761 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 21:11:19.501008 16761 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 21:11:19.504149 16761 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39885
I20260924 21:11:19.504182 16828 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39885 every 8 connection(s)
I20260924 21:11:19.504463 16761 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790284076800039-392-0/minicluster-data/master-0/data/info.pb
I20260924 21:11:19.505702 16829 sys_catalog.cc:263] Verifying existing consensus state
I20260924 21:11:19.506275 16829 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a18533bad310410ca2873fde2d8ff491: Bootstrap starting.
I20260924 21:11:19.507246 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAzNNjS/build/release/bin/kudu as pid 16761
I20260924 21:11:19.507530 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAzNNjS/build/release/bin/kudu
/tmp/dist-test-taskAzNNjS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790284076800039-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790284076800039-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790284076800039-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790284076800039-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:34291
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:39885
--webserver_port=39809
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790284076800039-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44143
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790284076800039-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790284076800039-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790284076800039-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 21:11:19.514226 16829 log.cc:824] T 00000000000000000000000000000000 P a18533bad310410ca2873fde2d8ff491: Log is configured to *not* fsync() on all Append() calls
I20260924 21:11:19.515991 16829 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a18533bad310410ca2873fde2d8ff491: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 21:11:19.516304 16829 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a18533bad310410ca2873fde2d8ff491: Bootstrap complete.
I20260924 21:11:19.518344 16829 raft_consensus.cc:348] T 00000000000000000000000000000000 P a18533bad310410ca2873fde2d8ff491 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a18533bad310410ca2873fde2d8ff491" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39885 } }
I20260924 21:11:19.518566 16829 raft_consensus.cc:729] T 00000000000000000000000000000000 P a18533bad310410ca2873fde2d8ff491 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: a18533bad310410ca2873fde2d8ff491, State: Initialized, Role: FOLLOWER
I20260924 21:11:19.518735 16829 consensus_queue.cc:260] T 00000000000000000000000000000000 P a18533bad310410ca2873fde2d8ff491 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a18533bad310410ca2873fde2d8ff491" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39885 } }
I20260924 21:11:19.518815 16829 raft_consensus.cc:388] T 00000000000000000000000000000000 P a18533bad310410ca2873fde2d8ff491 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 21:11:19.518862 16829 raft_consensus.cc:482] T 00000000000000000000000000000000 P a18533bad310410ca2873fde2d8ff491 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 21:11:19.518905 16829 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a18533bad310410ca2873fde2d8ff491 [term 2 FOLLOWER]: Advancing to term 3
I20260924 21:11:19.519726 16829 raft_consensus.cc:504] T 00000000000000000000000000000000 P a18533bad310410ca2873fde2d8ff491 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a18533bad310410ca2873fde2d8ff491" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39885 } }
I20260924 21:11:19.519821 16829 leader_election.cc:302] T 00000000000000000000000000000000 P a18533bad310410ca2873fde2d8ff491 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a18533bad310410ca2873fde2d8ff491; no voters:
I20260924 21:11:19.519979 16829 leader_election.cc:288] T 00000000000000000000000000000000 P a18533bad310410ca2873fde2d8ff491 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260924 21:11:19.520006 16833 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a18533bad310410ca2873fde2d8ff491 [term 3 FOLLOWER]: Leader election won for term 3
I20260924 21:11:19.520108 16833 raft_consensus.cc:686] T 00000000000000000000000000000000 P a18533bad310410ca2873fde2d8ff491 [term 3 LEADER]: Becoming Leader. State: Replica: a18533bad310410ca2873fde2d8ff491, State: Running, Role: LEADER
I20260924 21:11:19.520201 16829 sys_catalog.cc:564] T 00000000000000000000000000000000 P a18533bad310410ca2873fde2d8ff491 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 21:11:19.520237 16833 consensus_queue.cc:237] T 00000000000000000000000000000000 P a18533bad310410ca2873fde2d8ff491 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a18533bad310410ca2873fde2d8ff491" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39885 } }
I20260924 21:11:19.520473 16833 sys_catalog.cc:455] T 00000000000000000000000000000000 P a18533bad310410ca2873fde2d8ff491 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a18533bad310410ca2873fde2d8ff491. Latest consensus state: current_term: 3 leader_uuid: "a18533bad310410ca2873fde2d8ff491" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a18533bad310410ca2873fde2d8ff491" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39885 } } }
I20260924 21:11:19.520476 16834 sys_catalog.cc:455] T 00000000000000000000000000000000 P a18533bad310410ca2873fde2d8ff491 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "a18533bad310410ca2873fde2d8ff491" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a18533bad310410ca2873fde2d8ff491" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39885 } } }
I20260924 21:11:19.520756 16834 sys_catalog.cc:458] T 00000000000000000000000000000000 P a18533bad310410ca2873fde2d8ff491 [sys.catalog]: This master's current role is: LEADER
I20260924 21:11:19.520690 16833 sys_catalog.cc:458] T 00000000000000000000000000000000 P a18533bad310410ca2873fde2d8ff491 [sys.catalog]: This master's current role is: LEADER
I20260924 21:11:19.520944 16842 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 21:11:19.521677 16842 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 21:11:19.521814 16842 catalog_manager.cc:1259] Loaded cluster ID: 00ea91f509a845098739aef3b1bf5fff
I20260924 21:11:19.521894 16842 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 21:11:19.523026 16842 catalog_manager.cc:1504] Loading token signing keys...
I20260924 21:11:19.523176 16842 catalog_manager.cc:5879] T 00000000000000000000000000000000 P a18533bad310410ca2873fde2d8ff491: Loaded TSK: 0
I20260924 21:11:19.523212 16842 catalog_manager.cc:5879] T 00000000000000000000000000000000 P a18533bad310410ca2873fde2d8ff491: Loaded TSK: 1
F20260924 21:11:19.523399 16842 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f1084b8d9ad google::LogMessage::Fail() at ??:0
@ 0x7f1084b9088a google::LogMessage::SendToLog() at ??:0
@ 0x7f1084b8d4de google::LogMessage::Flush() at ??:0
@ 0x7f1084b8ea19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f10872fa104 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f10872f7d41 kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f108caeb1cd kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f108caeb48e kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f108caeb594 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7f108cab1c48 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7f108cabbad0 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f1086a5b0d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f1086a50954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f10878b46db start_thread at ??:0
@ 0x7f108368b61f clone at ??:0
W20260924 21:11:19.598098 16831 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 21:11:19.598259 16831 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 21:11:19.598281 16831 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 21:11:19.599468 16831 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 21:11:19.599514 16831 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 21:11:19.599552 16831 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 21:11:19.600994 16831 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44143
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790284076800039-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790284076800039-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790284076800039-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:34291
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790284076800039-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=39809
--tserver_master_addrs=127.0.98.62:39885
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16831
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790284076800039-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 19:49:06 UTC on ccycloud.quasar-vhqjoc.root.comops.site
build id 4670864
I20260924 21:11:19.601227 16831 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 21:11:19.601411 16831 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 21:11:19.605669 16870 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 21:11:19.605756 16831 server_base.cc:1034] running on GCE node
W20260924 21:11:19.605896 16873 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 21:11:19.605959 16871 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 21:11:19.606128 16831 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 21:11:19.606385 16831 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 21:11:19.607537 16831 hybrid_clock.cc:648] HybridClock initialized: now 1790284279607529 us; error 29 us; skew 500 ppm
Sep 24 21:11:19 dist-test-slave-c7tk krb5kdc[15694](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284279, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 21:11:19.609544 16831 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 21:11:19.609994 16831 webserver.cc:460] Webserver started at http://127.0.98.1:39809/ using document root <none> and password file <none>
I20260924 21:11:19.610220 16831 fs_manager.cc:359] Metadata directory not provided
I20260924 21:11:19.610261 16831 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 21:11:19.611393 16831 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 21:11:19.611985 16880 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 21:11:19.612124 16831 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 21:11:19.612196 16831 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790284076800039-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790284076800039-392-0/minicluster-data/ts-0/wal
uuid: "1d9aeb8d92ac439289b0193dee78ad72"
format_stamp: "Formatted at 2026-09-24 21:10:38 on dist-test-slave-c7tk"
server_key: "d72b83a7a68945cb826b8a2427e7a09f"
server_key_iv: "ac8be138b9e3bfd4dcf2a303f16243a0"
server_key_version: "encryptionkey@0"
I20260924 21:11:19.612465 16831 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790284076800039-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790284076800039-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790284076800039-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 21:11:19.627482 16831 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 21:11:19.629262 16831 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 21:11:19.629397 16831 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 21:11:19.629591 16831 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 21:11:19.630016 16887 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260924 21:11:19.631218 16831 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260924 21:11:19.631263 16831 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.001s user 0.000s sys 0.000s
I20260924 21:11:19.631289 16831 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260924 21:11:19.631834 16831 ts_tablet_manager.cc:610] Registered 1 tablets
I20260924 21:11:19.631871 16831 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260924 21:11:19.631908 16887 tablet_bootstrap.cc:492] T 33e360f293484359a0843fa409a630e3 P 1d9aeb8d92ac439289b0193dee78ad72: Bootstrap starting.
I20260924 21:11:19.638968 16831 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:34291
I20260924 21:11:19.639089 16994 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:34291 every 8 connection(s)
I20260924 21:11:19.639432 16831 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790284076800039-392-0/minicluster-data/ts-0/data/info.pb
I20260924 21:11:19.642354 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAzNNjS/build/release/bin/kudu as pid 16831
I20260924 21:11:19.642606 16887 log.cc:824] T 33e360f293484359a0843fa409a630e3 P 1d9aeb8d92ac439289b0193dee78ad72: Log is configured to *not* fsync() on all Append() calls
I20260924 21:11:19.642616 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAzNNjS/build/release/bin/kudu
/tmp/dist-test-taskAzNNjS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790284076800039-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790284076800039-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790284076800039-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790284076800039-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:38129
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:39885
--webserver_port=38699
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790284076800039-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44143
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790284076800039-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790284076800039-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790284076800039-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 21:11:19.644699 16887 tablet_bootstrap.cc:492] T 33e360f293484359a0843fa409a630e3 P 1d9aeb8d92ac439289b0193dee78ad72: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 21:11:19.644945 16887 tablet_bootstrap.cc:492] T 33e360f293484359a0843fa409a630e3 P 1d9aeb8d92ac439289b0193dee78ad72: Bootstrap complete.
I20260924 21:11:19.646023 16887 ts_tablet_manager.cc:1397] T 33e360f293484359a0843fa409a630e3 P 1d9aeb8d92ac439289b0193dee78ad72: Time spent bootstrapping tablet: real 0.014s user 0.007s sys 0.007s
I20260924 21:11:19.647800 16887 raft_consensus.cc:348] T 33e360f293484359a0843fa409a630e3 P 1d9aeb8d92ac439289b0193dee78ad72 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e99c21581cf14320a2db35bd3ba7e8c6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34643 } } peers { permanent_uuid: "1d9aeb8d92ac439289b0193dee78ad72" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34291 } } peers { permanent_uuid: "55193fd102fc483ab492f9b5a184b225" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38129 } }
I20260924 21:11:19.648156 16887 raft_consensus.cc:729] T 33e360f293484359a0843fa409a630e3 P 1d9aeb8d92ac439289b0193dee78ad72 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d9aeb8d92ac439289b0193dee78ad72, State: Initialized, Role: FOLLOWER
I20260924 21:11:19.648285 16887 consensus_queue.cc:260] T 33e360f293484359a0843fa409a630e3 P 1d9aeb8d92ac439289b0193dee78ad72 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e99c21581cf14320a2db35bd3ba7e8c6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34643 } } peers { permanent_uuid: "1d9aeb8d92ac439289b0193dee78ad72" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34291 } } peers { permanent_uuid: "55193fd102fc483ab492f9b5a184b225" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38129 } }
I20260924 21:11:19.648536 16887 ts_tablet_manager.cc:1428] T 33e360f293484359a0843fa409a630e3 P 1d9aeb8d92ac439289b0193dee78ad72: Time spent starting tablet: real 0.002s user 0.001s sys 0.002s
W20260924 21:11:19.695948 16997 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 21:11:19.640011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39885 (local address 127.0.98.1:37863)
0924 21:11:19.640158 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0924 21:11:19.640195 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0924 21:11:19.640846 (+ 651us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 21:11:19.695769 (+ 54923us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39885: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":86,"threads_started":1}
W20260924 21:11:19.696192 16995 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:39885 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:39885: Client connection negotiation failed: client connection to 127.0.98.62:39885: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260924 21:11:19.696554 16997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:11:19.696442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39885 (local address 127.0.98.1:36923)
0924 21:11:19.696491 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0924 21:11:19.696529 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20260924 21:11:19.696894 16997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:11:19.696804 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39885 (local address 127.0.98.1:48997)
0924 21:11:19.696850 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0924 21:11:19.696874 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260924 21:11:19.697001 16995 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260924 21:11:19.718534 16999 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 21:11:19.718715 16999 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 21:11:19.718747 16999 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 21:11:19.719703 16999 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 21:11:19.719756 16999 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 21:11:19.719794 16999 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 21:11:19.720976 16999 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44143
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790284076800039-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790284076800039-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790284076800039-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:38129
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790284076800039-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=38699
--tserver_master_addrs=127.0.98.62:39885
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16999
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790284076800039-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 19:49:06 UTC on ccycloud.quasar-vhqjoc.root.comops.site
build id 4670864
I20260924 21:11:19.721184 16999 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 21:11:19.721364 16999 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 21:11:19.725423 16999 server_base.cc:1034] running on GCE node
W20260924 21:11:19.725418 17006 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 21:11:19.725418 17005 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 21:11:19.725380 17008 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 21:11:19.725708 16999 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 21:11:19.725929 16999 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 21:11:19.727077 16999 hybrid_clock.cc:648] HybridClock initialized: now 1790284279727061 us; error 25 us; skew 500 ppm
Sep 24 21:11:19 dist-test-slave-c7tk krb5kdc[15694](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284279, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 21:11:19.729008 16999 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 21:11:19.729379 16999 webserver.cc:460] Webserver started at http://127.0.98.2:38699/ using document root <none> and password file <none>
I20260924 21:11:19.729552 16999 fs_manager.cc:359] Metadata directory not provided
I20260924 21:11:19.729605 16999 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 21:11:19.730692 16999 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.001s sys 0.000s
I20260924 21:11:19.731137 17015 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 21:11:19.731271 16999 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 21:11:19.731348 16999 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790284076800039-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790284076800039-392-0/minicluster-data/ts-1/wal
uuid: "55193fd102fc483ab492f9b5a184b225"
format_stamp: "Formatted at 2026-09-24 21:10:38 on dist-test-slave-c7tk"
server_key: "010dc19283e82651e052ef7c67606224"
server_key_iv: "d271153449d1e8c7437aa1571c4a91a5"
server_key_version: "encryptionkey@0"
I20260924 21:11:19.731611 16999 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790284076800039-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790284076800039-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790284076800039-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 21:11:19.739023 16999 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 21:11:19.740384 16999 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 21:11:19.740509 16999 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 21:11:19.740686 16999 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 21:11:19.740912 16999 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 21:11:19.740955 16999 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 21:11:19.740988 16999 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 21:11:19.741011 16999 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 21:11:19.746014 16999 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:38129
I20260924 21:11:19.746058 17128 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:38129 every 8 connection(s)
I20260924 21:11:19.746347 16999 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790284076800039-392-0/minicluster-data/ts-1/data/info.pb
I20260924 21:11:19.747216 17131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:11:19.746899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39885 (local address 127.0.98.2:50537)
0924 21:11:19.747024 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0924 21:11:19.747079 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94,"thread_start_us":63,"threads_started":1}
I20260924 21:11:19.747268 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAzNNjS/build/release/bin/kudu as pid 16999
I20260924 21:11:19.747553 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAzNNjS/build/release/bin/kudu
/tmp/dist-test-taskAzNNjS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790284076800039-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790284076800039-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790284076800039-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790284076800039-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:34643
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:39885
--webserver_port=44675
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790284076800039-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44143
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790284076800039-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790284076800039-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790284076800039-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 21:11:19.747581 17129 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:39885 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:39885: Client connection negotiation failed: client connection to 127.0.98.62:39885: connect: Connection refused (error 111)
I20260924 21:11:19.747839 17131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:11:19.747721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39885 (local address 127.0.98.2:54977)
0924 21:11:19.747798 (+ 77us) negotiation.cc:107] Waiting for socket to connect
0924 21:11:19.747822 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260924 21:11:19.748148 17131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:11:19.748068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39885 (local address 127.0.98.2:49119)
0924 21:11:19.748119 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0924 21:11:19.748136 (+ 17us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260924 21:11:19.748242 17129 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260924 21:11:19.818041 17132 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 21:11:19.818214 17132 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 21:11:19.818236 17132 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 21:11:19.819206 17132 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 21:11:19.819237 17132 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 21:11:19.819259 17132 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 21:11:19.820384 17132 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44143
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790284076800039-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790284076800039-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790284076800039-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:34643
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790284076800039-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=44675
--tserver_master_addrs=127.0.98.62:39885
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17132
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790284076800039-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 19:49:06 UTC on ccycloud.quasar-vhqjoc.root.comops.site
build id 4670864
I20260924 21:11:19.820521 17132 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 21:11:19.820650 17132 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 21:11:19.824946 17138 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 21:11:19.824932 17137 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 21:11:19.825148 17132 server_base.cc:1034] running on GCE node
W20260924 21:11:19.824934 17140 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 21:11:19.825372 17132 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 21:11:19.825641 17132 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 21:11:19.826771 17132 hybrid_clock.cc:648] HybridClock initialized: now 1790284279826740 us; error 45 us; skew 500 ppm
Sep 24 21:11:19 dist-test-slave-c7tk krb5kdc[15694](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284279, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 21:11:19.828840 17132 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 21:11:19.829227 17132 webserver.cc:460] Webserver started at http://127.0.98.3:44675/ using document root <none> and password file <none>
I20260924 21:11:19.829398 17132 fs_manager.cc:359] Metadata directory not provided
I20260924 21:11:19.829434 17132 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 21:11:19.830528 17132 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 21:11:19.830989 17147 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 21:11:19.831120 17132 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 21:11:19.831192 17132 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790284076800039-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790284076800039-392-0/minicluster-data/ts-2/wal
uuid: "e99c21581cf14320a2db35bd3ba7e8c6"
format_stamp: "Formatted at 2026-09-24 21:10:38 on dist-test-slave-c7tk"
server_key: "98c6f865964e69523d7f33892db0edea"
server_key_iv: "f396dd3f02cffa8e0103119e0154437a"
server_key_version: "encryptionkey@0"
I20260924 21:11:19.831409 17132 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790284076800039-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790284076800039-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790284076800039-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 21:11:19.852006 17132 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 21:11:19.853830 17132 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 21:11:19.853930 17132 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 21:11:19.854086 17132 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 21:11:19.854269 17132 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 21:11:19.854291 17132 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 21:11:19.854308 17132 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 21:11:19.854318 17132 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 21:11:19.859300 17132 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:34643
I20260924 21:11:19.859360 17260 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:34643 every 8 connection(s)
I20260924 21:11:19.859624 17132 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790284076800039-392-0/minicluster-data/ts-2/data/info.pb
I20260924 21:11:19.860416 17263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:11:19.860143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39885 (local address 127.0.98.3:40573)
0924 21:11:19.860264 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0924 21:11:19.860314 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93,"thread_start_us":58,"threads_started":1}
W20260924 21:11:19.860734 17261 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:39885 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:39885: Client connection negotiation failed: client connection to 127.0.98.62:39885: connect: Connection refused (error 111)
I20260924 21:11:19.860977 17263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:11:19.860892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39885 (local address 127.0.98.3:57855)
0924 21:11:19.860929 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0924 21:11:19.860951 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
I20260924 21:11:19.861217 17263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:11:19.861143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39885 (local address 127.0.98.3:34413)
0924 21:11:19.861187 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0924 21:11:19.861205 (+ 18us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260924 21:11:19.861331 17261 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260924 21:11:19.861665 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAzNNjS/build/release/bin/kudu as pid 17132
I20260924 21:11:20.697470 17265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:11:20.697272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39885 (local address 127.0.98.1:39813)
0924 21:11:20.697382 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0924 21:11:20.697412 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":54,"threads_started":1}
I20260924 21:11:20.748639 17266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:11:20.748487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39885 (local address 127.0.98.2:57125)
0924 21:11:20.748575 (+ 88us) negotiation.cc:107] Waiting for socket to connect
0924 21:11:20.748602 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76,"thread_start_us":45,"threads_started":1}
I20260924 21:11:20.861845 17267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:11:20.861659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39885 (local address 127.0.98.3:36139)
0924 21:11:20.861763 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0924 21:11:20.861792 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92,"thread_start_us":60,"threads_started":1}
I20260924 21:11:21.497468 17268 raft_consensus.cc:482] T 33e360f293484359a0843fa409a630e3 P 1d9aeb8d92ac439289b0193dee78ad72 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 21:11:21.497588 17268 raft_consensus.cc:504] T 33e360f293484359a0843fa409a630e3 P 1d9aeb8d92ac439289b0193dee78ad72 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e99c21581cf14320a2db35bd3ba7e8c6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34643 } } peers { permanent_uuid: "1d9aeb8d92ac439289b0193dee78ad72" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34291 } } peers { permanent_uuid: "55193fd102fc483ab492f9b5a184b225" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38129 } }
I20260924 21:11:21.497926 17268 leader_election.cc:288] T 33e360f293484359a0843fa409a630e3 P 1d9aeb8d92ac439289b0193dee78ad72 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e99c21581cf14320a2db35bd3ba7e8c6 (127.0.98.3:34643), 55193fd102fc483ab492f9b5a184b225 (127.0.98.2:38129)
Sep 24 21:11:21 dist-test-slave-c7tk krb5kdc[15694](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284279, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.3@KRBTEST.COM
Sep 24 21:11:21 dist-test-slave-c7tk krb5kdc[15694](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284279, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.2@KRBTEST.COM
I20260924 21:11:21.507264 17269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:11:21.498058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:34643 (local address 127.0.98.1:46383)
0924 21:11:21.498298 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0924 21:11:21.498309 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 21:11:21.498367 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 21:11:21.499023 (+ 656us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 21:11:21.499026 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 21:11:21.499418 (+ 392us) client_negotiation.cc:189] Negotiated authn=SASL
0924 21:11:21.499597 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:11:21.499604 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:11:21.500549 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:11:21.500549 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:11:21.501251 (+ 702us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:11:21.501253 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:11:21.501749 (+ 496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:11:21.501751 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:11:21.501887 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:11:21.502403 (+ 516us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 21:11:21.502412 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 21:11:21.503958 (+ 1546us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 21:11:21.505717 (+ 1759us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 21:11:21.505723 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 21:11:21.505725 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 21:11:21.506078 (+ 353us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 21:11:21.506306 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 21:11:21.506307 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 21:11:21.506308 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 21:11:21.506423 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 21:11:21.506767 (+ 344us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 21:11:21.506773 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 21:11:21.506914 (+ 141us) client_negotiation.cc:769] Sending connection context
0924 21:11:21.507003 (+ 89us) client_negotiation.cc:240] Negotiation successful
0924 21:11:21.507165 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":224,"mutex_wait_us":32,"thread_start_us":81,"threads_started":1}
I20260924 21:11:21.507283 17270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:11:21.498163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:46383 (local address 127.0.98.3:34643)
0924 21:11:21.498292 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0924 21:11:21.498304 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 21:11:21.498362 (+ 58us) server_negotiation.cc:407] Connection header received
0924 21:11:21.498837 (+ 475us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:11:21.498839 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:11:21.498871 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:11:21.498996 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:11:21.499718 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:11:21.500444 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:11:21.501333 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:11:21.501674 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:11:21.501745 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:11:21.504057 (+ 2312us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 21:11:21.504061 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 21:11:21.504061 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 21:11:21.504071 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0924 21:11:21.505645 (+ 1574us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:11:21.506157 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:11:21.506160 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:11:21.506161 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:11:21.506245 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:11:21.506490 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:11:21.506491 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:11:21.506492 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:11:21.506698 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 21:11:21.506785 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0924 21:11:21.507062 (+ 277us) server_negotiation.cc:299] Negotiation successful
0924 21:11:21.507186 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93,"thread_start_us":51,"threads_started":1}
I20260924 21:11:21.507309 17272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:11:21.498327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:53117 (local address 127.0.98.2:38129)
0924 21:11:21.498431 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0924 21:11:21.498437 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 21:11:21.498457 (+ 20us) server_negotiation.cc:407] Connection header received
0924 21:11:21.498830 (+ 373us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:11:21.498832 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:11:21.498868 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:11:21.498999 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:11:21.499737 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:11:21.500451 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:11:21.501845 (+ 1394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:11:21.502235 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:11:21.502316 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:11:21.504298 (+ 1982us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 21:11:21.504301 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 21:11:21.504301 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 21:11:21.504309 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 21:11:21.505739 (+ 1430us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:11:21.506181 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:11:21.506183 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:11:21.506183 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:11:21.506261 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:11:21.506486 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:11:21.506487 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:11:21.506488 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:11:21.506749 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 21:11:21.506833 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0924 21:11:21.507049 (+ 216us) server_negotiation.cc:299] Negotiation successful
0924 21:11:21.507195 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77,"thread_start_us":49,"threads_started":1}
I20260924 21:11:21.507373 17271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:11:21.498235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38129 (local address 127.0.98.1:53117)
0924 21:11:21.498312 (+ 77us) negotiation.cc:107] Waiting for socket to connect
0924 21:11:21.498320 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0924 21:11:21.498364 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 21:11:21.499023 (+ 659us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 21:11:21.499026 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 21:11:21.499426 (+ 400us) client_negotiation.cc:189] Negotiated authn=SASL
0924 21:11:21.499605 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:11:21.499608 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:11:21.500539 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:11:21.500540 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:11:21.501253 (+ 713us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:11:21.501255 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:11:21.502313 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:11:21.502314 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:11:21.502370 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:11:21.502646 (+ 276us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 21:11:21.502650 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 21:11:21.504213 (+ 1563us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 21:11:21.505874 (+ 1661us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 21:11:21.505875 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 21:11:21.505875 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 21:11:21.506089 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 21:11:21.506339 (+ 250us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 21:11:21.506340 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 21:11:21.506341 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 21:11:21.506423 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 21:11:21.506829 (+ 406us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 21:11:21.506829 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 21:11:21.506904 (+ 75us) client_negotiation.cc:769] Sending connection context
0924 21:11:21.506997 (+ 93us) client_negotiation.cc:240] Negotiation successful
0924 21:11:21.507239 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":70,"mutex_wait_us":150,"thread_start_us":43,"threads_started":1}
W20260924 21:11:21.508232 16882 leader_election.cc:341] T 33e360f293484359a0843fa409a630e3 P 1d9aeb8d92ac439289b0193dee78ad72 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e99c21581cf14320a2db35bd3ba7e8c6 (127.0.98.3:34643): Not found: Tablet not found: 33e360f293484359a0843fa409a630e3
W20260924 21:11:21.508293 16882 leader_election.cc:341] T 33e360f293484359a0843fa409a630e3 P 1d9aeb8d92ac439289b0193dee78ad72 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 55193fd102fc483ab492f9b5a184b225 (127.0.98.2:38129): Not found: Tablet not found: 33e360f293484359a0843fa409a630e3
I20260924 21:11:21.508314 16882 leader_election.cc:302] T 33e360f293484359a0843fa409a630e3 P 1d9aeb8d92ac439289b0193dee78ad72 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1d9aeb8d92ac439289b0193dee78ad72; no voters: 55193fd102fc483ab492f9b5a184b225, e99c21581cf14320a2db35bd3ba7e8c6
I20260924 21:11:21.508394 17268 raft_consensus.cc:2726] T 33e360f293484359a0843fa409a630e3 P 1d9aeb8d92ac439289b0193dee78ad72 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 21:11:21.698062 17271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:11:21.697947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39885 (local address 127.0.98.1:37067)
0924 21:11:21.697992 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0924 21:11:21.698023 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
I20260924 21:11:21.749195 17273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:11:21.748997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39885 (local address 127.0.98.2:47799)
0924 21:11:21.749111 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0924 21:11:21.749155 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":48,"threads_started":1}
I20260924 21:11:21.862411 17274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:11:21.862203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39885 (local address 127.0.98.3:41807)
0924 21:11:21.862331 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0924 21:11:21.862373 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":68,"threads_started":1}
I20260924 21:11:22.698645 17275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:11:22.698445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39885 (local address 127.0.98.1:37411)
0924 21:11:22.698552 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0924 21:11:22.698582 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93,"thread_start_us":58,"threads_started":1}
I20260924 21:11:22.749883 17276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:11:22.749575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39885 (local address 127.0.98.2:53613)
0924 21:11:22.749774 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0924 21:11:22.749845 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":58,"threads_started":1}
I20260924 21:11:22.862928 17277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:11:22.862769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39885 (local address 127.0.98.3:60423)
0924 21:11:22.862868 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0924 21:11:22.862896 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86,"thread_start_us":56,"threads_started":1}
I20260924 21:11:23.204021 17278 raft_consensus.cc:482] T 33e360f293484359a0843fa409a630e3 P 1d9aeb8d92ac439289b0193dee78ad72 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 21:11:23.204126 17278 raft_consensus.cc:504] T 33e360f293484359a0843fa409a630e3 P 1d9aeb8d92ac439289b0193dee78ad72 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e99c21581cf14320a2db35bd3ba7e8c6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34643 } } peers { permanent_uuid: "1d9aeb8d92ac439289b0193dee78ad72" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34291 } } peers { permanent_uuid: "55193fd102fc483ab492f9b5a184b225" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38129 } }
I20260924 21:11:23.204275 17278 leader_election.cc:288] T 33e360f293484359a0843fa409a630e3 P 1d9aeb8d92ac439289b0193dee78ad72 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e99c21581cf14320a2db35bd3ba7e8c6 (127.0.98.3:34643), 55193fd102fc483ab492f9b5a184b225 (127.0.98.2:38129)
W20260924 21:11:23.204627 16882 leader_election.cc:341] T 33e360f293484359a0843fa409a630e3 P 1d9aeb8d92ac439289b0193dee78ad72 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e99c21581cf14320a2db35bd3ba7e8c6 (127.0.98.3:34643): Not found: Tablet not found: 33e360f293484359a0843fa409a630e3
W20260924 21:11:23.204718 16882 leader_election.cc:341] T 33e360f293484359a0843fa409a630e3 P 1d9aeb8d92ac439289b0193dee78ad72 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 55193fd102fc483ab492f9b5a184b225 (127.0.98.2:38129): Not found: Tablet not found: 33e360f293484359a0843fa409a630e3
I20260924 21:11:23.204752 16882 leader_election.cc:302] T 33e360f293484359a0843fa409a630e3 P 1d9aeb8d92ac439289b0193dee78ad72 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1d9aeb8d92ac439289b0193dee78ad72; no voters: 55193fd102fc483ab492f9b5a184b225, e99c21581cf14320a2db35bd3ba7e8c6
I20260924 21:11:23.204833 17278 raft_consensus.cc:2726] T 33e360f293484359a0843fa409a630e3 P 1d9aeb8d92ac439289b0193dee78ad72 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 21:11:23.699399 17279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:11:23.699191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39885 (local address 127.0.98.1:38353)
0924 21:11:23.699320 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0924 21:11:23.699350 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":81,"threads_started":1}
I20260924 21:11:23.750490 17280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:11:23.750282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39885 (local address 127.0.98.2:40163)
0924 21:11:23.750400 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0924 21:11:23.750436 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":54,"threads_started":1}
I20260924 21:11:23.863483 17281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:11:23.863268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39885 (local address 127.0.98.3:39019)
0924 21:11:23.863380 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0924 21:11:23.863410 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":65,"threads_started":1}
I20260924 21:11:24.700124 17282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:11:24.699924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39885 (local address 127.0.98.1:52727)
0924 21:11:24.700052 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0924 21:11:24.700080 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":80,"threads_started":1}
I20260924 21:11:24.751109 17283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:11:24.750947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39885 (local address 127.0.98.2:38919)
0924 21:11:24.751053 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0924 21:11:24.751083 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94,"thread_start_us":51,"threads_started":1}
I20260924 21:11:24.864142 17284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:11:24.863976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39885 (local address 127.0.98.3:42481)
0924 21:11:24.864082 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0924 21:11:24.864112 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93,"thread_start_us":60,"threads_started":1}
I20260924 21:11:25.700814 17285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:11:25.700632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39885 (local address 127.0.98.1:56561)
0924 21:11:25.700740 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0924 21:11:25.700770 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":57,"threads_started":1}
I20260924 21:11:25.751770 17286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:11:25.751601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39885 (local address 127.0.98.2:43629)
0924 21:11:25.751712 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0924 21:11:25.751742 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":57,"threads_started":1}
I20260924 21:11:25.864666 17287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:11:25.864508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39885 (local address 127.0.98.3:48103)
0924 21:11:25.864607 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0924 21:11:25.864635 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87,"thread_start_us":48,"threads_started":1}
I20260924 21:11:26.153105 17288 raft_consensus.cc:482] T 33e360f293484359a0843fa409a630e3 P 1d9aeb8d92ac439289b0193dee78ad72 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 21:11:26.153198 17288 raft_consensus.cc:504] T 33e360f293484359a0843fa409a630e3 P 1d9aeb8d92ac439289b0193dee78ad72 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e99c21581cf14320a2db35bd3ba7e8c6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34643 } } peers { permanent_uuid: "1d9aeb8d92ac439289b0193dee78ad72" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34291 } } peers { permanent_uuid: "55193fd102fc483ab492f9b5a184b225" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38129 } }
I20260924 21:11:26.153338 17288 leader_election.cc:288] T 33e360f293484359a0843fa409a630e3 P 1d9aeb8d92ac439289b0193dee78ad72 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e99c21581cf14320a2db35bd3ba7e8c6 (127.0.98.3:34643), 55193fd102fc483ab492f9b5a184b225 (127.0.98.2:38129)
W20260924 21:11:26.153869 16882 leader_election.cc:341] T 33e360f293484359a0843fa409a630e3 P 1d9aeb8d92ac439289b0193dee78ad72 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e99c21581cf14320a2db35bd3ba7e8c6 (127.0.98.3:34643): Not found: Tablet not found: 33e360f293484359a0843fa409a630e3
W20260924 21:11:26.153946 16882 leader_election.cc:341] T 33e360f293484359a0843fa409a630e3 P 1d9aeb8d92ac439289b0193dee78ad72 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 55193fd102fc483ab492f9b5a184b225 (127.0.98.2:38129): Not found: Tablet not found: 33e360f293484359a0843fa409a630e3
I20260924 21:11:26.153961 16882 leader_election.cc:302] T 33e360f293484359a0843fa409a630e3 P 1d9aeb8d92ac439289b0193dee78ad72 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1d9aeb8d92ac439289b0193dee78ad72; no voters: 55193fd102fc483ab492f9b5a184b225, e99c21581cf14320a2db35bd3ba7e8c6
I20260924 21:11:26.154054 17288 raft_consensus.cc:2726] T 33e360f293484359a0843fa409a630e3 P 1d9aeb8d92ac439289b0193dee78ad72 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 21:11:26.701502 17289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:11:26.701300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39885 (local address 127.0.98.1:56981)
0924 21:11:26.701422 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0924 21:11:26.701456 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":53,"threads_started":1}
I20260924 21:11:26.752393 17290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:11:26.752233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39885 (local address 127.0.98.2:51273)
0924 21:11:26.752329 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0924 21:11:26.752363 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82,"thread_start_us":48,"threads_started":1}
I20260924 21:11:26.865206 17291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:11:26.865026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39885 (local address 127.0.98.3:34313)
0924 21:11:26.865143 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0924 21:11:26.865175 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":51,"threads_started":1}
I20260924 21:11:27.702103 17292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:11:27.701923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39885 (local address 127.0.98.1:60221)
0924 21:11:27.702035 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0924 21:11:27.702061 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92,"thread_start_us":64,"threads_started":1}
I20260924 21:11:27.752868 17293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:11:27.752710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39885 (local address 127.0.98.2:60509)
0924 21:11:27.752809 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0924 21:11:27.752834 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89,"thread_start_us":57,"threads_started":1}
I20260924 21:11:27.865906 17294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:11:27.865619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39885 (local address 127.0.98.3:47497)
0924 21:11:27.865839 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0924 21:11:27.865872 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":64,"threads_started":1}
I20260924 21:11:28.702687 17295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:11:28.702496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39885 (local address 127.0.98.1:48249)
0924 21:11:28.702610 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0924 21:11:28.702642 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":51,"threads_started":1}
I20260924 21:11:28.753371 17296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:11:28.753198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39885 (local address 127.0.98.2:46799)
0924 21:11:28.753306 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0924 21:11:28.753339 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":48,"threads_started":1}
I20260924 21:11:28.866446 17297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:11:28.866274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39885 (local address 127.0.98.3:53791)
0924 21:11:28.866381 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0924 21:11:28.866412 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93,"thread_start_us":50,"threads_started":1}
I20260924 21:11:29.703359 17298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:11:29.703174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39885 (local address 127.0.98.1:48135)
0924 21:11:29.703289 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0924 21:11:29.703317 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":68,"threads_started":1}
I20260924 21:11:29.753997 17299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:11:29.753764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39885 (local address 127.0.98.2:54955)
0924 21:11:29.753937 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0924 21:11:29.753964 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":119,"threads_started":1}
I20260924 21:11:29.867081 17300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:11:29.866868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39885 (local address 127.0.98.3:56723)
0924 21:11:29.866988 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0924 21:11:29.867022 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":63,"threads_started":1}
I20260924 21:11:30.255941 17301 raft_consensus.cc:482] T 33e360f293484359a0843fa409a630e3 P 1d9aeb8d92ac439289b0193dee78ad72 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 21:11:30.256042 17301 raft_consensus.cc:504] T 33e360f293484359a0843fa409a630e3 P 1d9aeb8d92ac439289b0193dee78ad72 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e99c21581cf14320a2db35bd3ba7e8c6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34643 } } peers { permanent_uuid: "1d9aeb8d92ac439289b0193dee78ad72" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34291 } } peers { permanent_uuid: "55193fd102fc483ab492f9b5a184b225" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38129 } }
I20260924 21:11:30.256206 17301 leader_election.cc:288] T 33e360f293484359a0843fa409a630e3 P 1d9aeb8d92ac439289b0193dee78ad72 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e99c21581cf14320a2db35bd3ba7e8c6 (127.0.98.3:34643), 55193fd102fc483ab492f9b5a184b225 (127.0.98.2:38129)
W20260924 21:11:30.256556 16882 leader_election.cc:341] T 33e360f293484359a0843fa409a630e3 P 1d9aeb8d92ac439289b0193dee78ad72 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e99c21581cf14320a2db35bd3ba7e8c6 (127.0.98.3:34643): Not found: Tablet not found: 33e360f293484359a0843fa409a630e3
W20260924 21:11:30.256654 16882 leader_election.cc:341] T 33e360f293484359a0843fa409a630e3 P 1d9aeb8d92ac439289b0193dee78ad72 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 55193fd102fc483ab492f9b5a184b225 (127.0.98.2:38129): Not found: Tablet not found: 33e360f293484359a0843fa409a630e3
I20260924 21:11:30.256685 16882 leader_election.cc:302] T 33e360f293484359a0843fa409a630e3 P 1d9aeb8d92ac439289b0193dee78ad72 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1d9aeb8d92ac439289b0193dee78ad72; no voters: 55193fd102fc483ab492f9b5a184b225, e99c21581cf14320a2db35bd3ba7e8c6
I20260924 21:11:30.256737 17301 raft_consensus.cc:2726] T 33e360f293484359a0843fa409a630e3 P 1d9aeb8d92ac439289b0193dee78ad72 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 21:11:30.703912 17302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:11:30.703729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39885 (local address 127.0.98.1:55367)
0924 21:11:30.703843 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0924 21:11:30.703869 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":69,"threads_started":1}
I20260924 21:11:30.754534 17303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:11:30.754354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39885 (local address 127.0.98.2:59857)
0924 21:11:30.754463 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0924 21:11:30.754491 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":61,"threads_started":1}
I20260924 21:11:30.867697 17304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:11:30.867506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39885 (local address 127.0.98.3:52851)
0924 21:11:30.867636 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0924 21:11:30.867663 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":78,"threads_started":1}
I20260924 21:11:31.704543 17305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:11:31.704265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39885 (local address 127.0.98.1:38661)
0924 21:11:31.704468 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0924 21:11:31.704502 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":62,"threads_started":1}
I20260924 21:11:31.755139 17306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:11:31.754930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39885 (local address 127.0.98.2:59851)
0924 21:11:31.755077 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0924 21:11:31.755108 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":75,"threads_started":1}
I20260924 21:11:31.868230 17307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:11:31.868030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39885 (local address 127.0.98.3:60763)
0924 21:11:31.868161 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0924 21:11:31.868195 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":56,"threads_started":1}
I20260924 21:11:32.705199 17308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:11:32.704983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39885 (local address 127.0.98.1:56007)
0924 21:11:32.705129 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0924 21:11:32.705156 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":68,"threads_started":1}
I20260924 21:11:32.755724 17309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:11:32.755519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39885 (local address 127.0.98.2:60915)
0924 21:11:32.755622 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0924 21:11:32.755660 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86,"thread_start_us":54,"threads_started":1}
I20260924 21:11:32.868932 17310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:11:32.868736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39885 (local address 127.0.98.3:35409)
0924 21:11:32.868861 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0924 21:11:32.868895 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":54,"threads_started":1}
I20260924 21:11:33.256726 17311 raft_consensus.cc:482] T 33e360f293484359a0843fa409a630e3 P 1d9aeb8d92ac439289b0193dee78ad72 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 21:11:33.256799 17311 raft_consensus.cc:504] T 33e360f293484359a0843fa409a630e3 P 1d9aeb8d92ac439289b0193dee78ad72 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e99c21581cf14320a2db35bd3ba7e8c6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34643 } } peers { permanent_uuid: "1d9aeb8d92ac439289b0193dee78ad72" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34291 } } peers { permanent_uuid: "55193fd102fc483ab492f9b5a184b225" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38129 } }
I20260924 21:11:33.256908 17311 leader_election.cc:288] T 33e360f293484359a0843fa409a630e3 P 1d9aeb8d92ac439289b0193dee78ad72 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e99c21581cf14320a2db35bd3ba7e8c6 (127.0.98.3:34643), 55193fd102fc483ab492f9b5a184b225 (127.0.98.2:38129)
W20260924 21:11:33.257287 16882 leader_election.cc:341] T 33e360f293484359a0843fa409a630e3 P 1d9aeb8d92ac439289b0193dee78ad72 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e99c21581cf14320a2db35bd3ba7e8c6 (127.0.98.3:34643): Not found: Tablet not found: 33e360f293484359a0843fa409a630e3
W20260924 21:11:33.257346 16882 leader_election.cc:341] T 33e360f293484359a0843fa409a630e3 P 1d9aeb8d92ac439289b0193dee78ad72 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 55193fd102fc483ab492f9b5a184b225 (127.0.98.2:38129): Not found: Tablet not found: 33e360f293484359a0843fa409a630e3
I20260924 21:11:33.257385 16882 leader_election.cc:302] T 33e360f293484359a0843fa409a630e3 P 1d9aeb8d92ac439289b0193dee78ad72 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1d9aeb8d92ac439289b0193dee78ad72; no voters: 55193fd102fc483ab492f9b5a184b225, e99c21581cf14320a2db35bd3ba7e8c6
I20260924 21:11:33.257459 17311 raft_consensus.cc:2726] T 33e360f293484359a0843fa409a630e3 P 1d9aeb8d92ac439289b0193dee78ad72 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 21:11:33.705789 17312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:11:33.705612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39885 (local address 127.0.98.1:56549)
0924 21:11:33.705718 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0924 21:11:33.705747 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92,"thread_start_us":51,"threads_started":1}
I20260924 21:11:33.756306 17313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:11:33.756073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39885 (local address 127.0.98.2:49153)
0924 21:11:33.756245 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0924 21:11:33.756275 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":45,"threads_started":1}
I20260924 21:11:33.869498 17314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:11:33.869324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39885 (local address 127.0.98.3:38427)
0924 21:11:33.869434 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0924 21:11:33.869467 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":50,"threads_started":1}
I20260924 21:11:34.706458 17315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:11:34.706262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39885 (local address 127.0.98.1:48543)
0924 21:11:34.706388 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0924 21:11:34.706417 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":71,"threads_started":1}
I20260924 21:11:34.756899 17316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:11:34.756735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39885 (local address 127.0.98.2:59117)
0924 21:11:34.756836 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0924 21:11:34.756865 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89,"thread_start_us":58,"threads_started":1}
I20260924 21:11:34.865831 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAzNNjS/build/release/bin/kudu with pid 16831
I20260924 21:11:34.870253 17317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:11:34.870056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39885 (local address 127.0.98.3:40427)
0924 21:11:34.870162 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0924 21:11:34.870211 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91,"thread_start_us":57,"threads_started":1}
I20260924 21:11:34.870596 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAzNNjS/build/release/bin/kudu with pid 16999
I20260924 21:11:34.874962 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAzNNjS/build/release/bin/kudu with pid 17132
2026-09-24T21:11:34Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (57870 ms)
[----------] 30 tests from SecurityITest (218083 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790284076800039-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 21:11:34 dist-test-slave-c7tk krb5kdc[17320](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 21:11:34 dist-test-slave-c7tk krb5kdc[17320](info): set up 2 sockets
Sep 24 21:11:34 dist-test-slave-c7tk krb5kdc[17320](info): commencing operation
krb5kdc: starting...
W20260924 21:11:35.813006 392 mini_kdc.cc:121] Time spent starting KDC: real 0.917s user 0.000s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790284076800039-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790284076800039-392-0/krb5kdc/test-user.keytab.
Sep 24 21:11:35 dist-test-slave-c7tk krb5kdc[17320](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284295, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T21:11:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T21:11:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790284076800039-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790284076800039-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 21:11:35.912494 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAzNNjS/build/release/bin/kudu
/tmp/dist-test-taskAzNNjS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790284076800039-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790284076800039-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790284076800039-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790284076800039-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34807
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790284076800039-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45229
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34807
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790284076800039-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790284076800039-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790284076800039-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 21:11:35.980635 17333 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 21:11:35.980824 17333 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 21:11:35.980854 17333 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 21:11:35.982232 17333 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 21:11:35.982285 17333 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 21:11:35.982307 17333 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 21:11:35.982331 17333 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 21:11:35.982354 17333 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 21:11:35.983944 17333 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45229
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790284076800039-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790284076800039-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34807
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790284076800039-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34807
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790284076800039-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17333
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790284076800039-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 19:49:06 UTC on ccycloud.quasar-vhqjoc.root.comops.site
build id 4670864
I20260924 21:11:35.984177 17333 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 21:11:35.984400 17333 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 21:11:35.988898 17333 server_base.cc:1034] running on GCE node
W20260924 21:11:35.988883 17341 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 21:11:35.988888 17339 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 21:11:35.988881 17338 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 21:11:35.989284 17333 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 21:11:35.989477 17333 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 21:11:35.990612 17333 hybrid_clock.cc:648] HybridClock initialized: now 1790284295990581 us; error 54 us; skew 500 ppm
Sep 24 21:11:35 dist-test-slave-c7tk krb5kdc[17320](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284295, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 21:11:35.992690 17333 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 21:11:35.993120 17333 webserver.cc:460] Webserver started at http://127.0.98.62:46123/ using document root <none> and password file <none>
I20260924 21:11:35.993314 17333 fs_manager.cc:359] Metadata directory not provided
I20260924 21:11:35.993374 17333 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 21:11:35.993520 17333 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 21:11:35.994391 17333 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790284076800039-392-0/minicluster-data/master-0/data/instance:
uuid: "f5608d721f6d49918c448489f906e37e"
format_stamp: "Formatted at 2026-09-24 21:11:35 on dist-test-slave-c7tk"
server_key: "5995507642820cc763ea4d78c37801b7"
server_key_iv: "564cc4dd42a44c510255b7f9443715a4"
server_key_version: "encryptionkey@0"
I20260924 21:11:35.994683 17333 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790284076800039-392-0/minicluster-data/master-0/wal/instance:
uuid: "f5608d721f6d49918c448489f906e37e"
format_stamp: "Formatted at 2026-09-24 21:11:35 on dist-test-slave-c7tk"
server_key: "5995507642820cc763ea4d78c37801b7"
server_key_iv: "564cc4dd42a44c510255b7f9443715a4"
server_key_version: "encryptionkey@0"
I20260924 21:11:35.995769 17333 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260924 21:11:35.996456 17348 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 21:11:35.996618 17333 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 21:11:35.996712 17333 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790284076800039-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790284076800039-392-0/minicluster-data/master-0/wal
uuid: "f5608d721f6d49918c448489f906e37e"
format_stamp: "Formatted at 2026-09-24 21:11:35 on dist-test-slave-c7tk"
server_key: "5995507642820cc763ea4d78c37801b7"
server_key_iv: "564cc4dd42a44c510255b7f9443715a4"
server_key_version: "encryptionkey@0"
I20260924 21:11:35.996789 17333 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790284076800039-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790284076800039-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790284076800039-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 21:11:36.021363 17333 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 21:11:36.023027 17333 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 21:11:36.023176 17333 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 21:11:36.027128 17333 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34807
I20260924 21:11:36.027163 17400 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34807 every 8 connection(s)
I20260924 21:11:36.027468 17333 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790284076800039-392-0/minicluster-data/master-0/data/info.pb
I20260924 21:11:36.028241 17401 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 21:11:36.030185 17401 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f5608d721f6d49918c448489f906e37e: Bootstrap starting.
I20260924 21:11:36.030653 17401 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f5608d721f6d49918c448489f906e37e: Neither blocks nor log segments found. Creating new log.
I20260924 21:11:36.030897 17401 log.cc:824] T 00000000000000000000000000000000 P f5608d721f6d49918c448489f906e37e: Log is configured to *not* fsync() on all Append() calls
I20260924 21:11:36.031510 17401 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f5608d721f6d49918c448489f906e37e: No bootstrap required, opened a new log
I20260924 21:11:36.032855 17401 raft_consensus.cc:348] T 00000000000000000000000000000000 P f5608d721f6d49918c448489f906e37e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5608d721f6d49918c448489f906e37e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34807 } }
I20260924 21:11:36.032965 17401 raft_consensus.cc:374] T 00000000000000000000000000000000 P f5608d721f6d49918c448489f906e37e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 21:11:36.032985 17401 raft_consensus.cc:729] T 00000000000000000000000000000000 P f5608d721f6d49918c448489f906e37e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f5608d721f6d49918c448489f906e37e, State: Initialized, Role: FOLLOWER
I20260924 21:11:36.033134 17401 consensus_queue.cc:260] T 00000000000000000000000000000000 P f5608d721f6d49918c448489f906e37e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5608d721f6d49918c448489f906e37e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34807 } }
I20260924 21:11:36.033190 17401 raft_consensus.cc:388] T 00000000000000000000000000000000 P f5608d721f6d49918c448489f906e37e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 21:11:36.033208 17401 raft_consensus.cc:482] T 00000000000000000000000000000000 P f5608d721f6d49918c448489f906e37e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 21:11:36.033263 17401 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f5608d721f6d49918c448489f906e37e [term 0 FOLLOWER]: Advancing to term 1
I20260924 21:11:36.033767 17401 raft_consensus.cc:504] T 00000000000000000000000000000000 P f5608d721f6d49918c448489f906e37e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5608d721f6d49918c448489f906e37e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34807 } }
I20260924 21:11:36.033924 17401 leader_election.cc:302] T 00000000000000000000000000000000 P f5608d721f6d49918c448489f906e37e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f5608d721f6d49918c448489f906e37e; no voters:
I20260924 21:11:36.034106 17401 leader_election.cc:288] T 00000000000000000000000000000000 P f5608d721f6d49918c448489f906e37e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 21:11:36.034163 17404 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f5608d721f6d49918c448489f906e37e [term 1 FOLLOWER]: Leader election won for term 1
I20260924 21:11:36.034343 17404 raft_consensus.cc:686] T 00000000000000000000000000000000 P f5608d721f6d49918c448489f906e37e [term 1 LEADER]: Becoming Leader. State: Replica: f5608d721f6d49918c448489f906e37e, State: Running, Role: LEADER
I20260924 21:11:36.034361 17401 sys_catalog.cc:564] T 00000000000000000000000000000000 P f5608d721f6d49918c448489f906e37e [sys.catalog]: configured and running, proceeding with master startup.
I20260924 21:11:36.034518 17404 consensus_queue.cc:237] T 00000000000000000000000000000000 P f5608d721f6d49918c448489f906e37e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5608d721f6d49918c448489f906e37e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34807 } }
I20260924 21:11:36.034914 17406 sys_catalog.cc:455] T 00000000000000000000000000000000 P f5608d721f6d49918c448489f906e37e [sys.catalog]: SysCatalogTable state changed. Reason: New leader f5608d721f6d49918c448489f906e37e. Latest consensus state: current_term: 1 leader_uuid: "f5608d721f6d49918c448489f906e37e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5608d721f6d49918c448489f906e37e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34807 } } }
I20260924 21:11:36.035032 17405 sys_catalog.cc:455] T 00000000000000000000000000000000 P f5608d721f6d49918c448489f906e37e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f5608d721f6d49918c448489f906e37e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5608d721f6d49918c448489f906e37e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34807 } } }
I20260924 21:11:36.035116 17405 sys_catalog.cc:458] T 00000000000000000000000000000000 P f5608d721f6d49918c448489f906e37e [sys.catalog]: This master's current role is: LEADER
I20260924 21:11:36.035045 17406 sys_catalog.cc:458] T 00000000000000000000000000000000 P f5608d721f6d49918c448489f906e37e [sys.catalog]: This master's current role is: LEADER
I20260924 21:11:36.035410 17413 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 21:11:36.035856 17413 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 21:11:36.036810 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAzNNjS/build/release/bin/kudu as pid 17333
I20260924 21:11:36.036896 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790284076800039-392-0/minicluster-data/master-0/wal/instance
I20260924 21:11:36.037091 392 external_mini_cluster.cc:1428] Setting key 73bf7a5c68a826ed49c06752e9522b9d
I20260924 21:11:36.037320 17413 catalog_manager.cc:1347] Generated new cluster ID: 466e07eaab984fafa57e2ae9f7f5fc87
I20260924 21:11:36.037380 17413 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 24 21:11:36 dist-test-slave-c7tk krb5kdc[17320](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284295, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 21:11:36.048790 17413 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 21:11:36.049358 17413 catalog_manager.cc:1504] Loading token signing keys...
I20260924 21:11:36.051913 17424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:11:36.037946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44540 (local address 127.0.98.62:34807)
0924 21:11:36.041786 (+ 3840us) server_negotiation.cc:206] Beginning negotiation
0924 21:11:36.041813 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0924 21:11:36.041847 (+ 34us) server_negotiation.cc:407] Connection header received
0924 21:11:36.041967 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:11:36.041974 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:11:36.042029 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:11:36.042152 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:11:36.042903 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:11:36.044114 (+ 1211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:11:36.045072 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:11:36.045663 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:11:36.045763 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:11:36.048424 (+ 2661us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 21:11:36.048428 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 21:11:36.048428 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 21:11:36.048442 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0924 21:11:36.050264 (+ 1822us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:11:36.050608 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:11:36.050610 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:11:36.050611 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:11:36.050695 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:11:36.050858 (+ 163us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:11:36.050859 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:11:36.050859 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:11:36.051069 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 21:11:36.051184 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0924 21:11:36.051605 (+ 421us) server_negotiation.cc:299] Negotiation successful
0924 21:11:36.051726 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3794,"thread_start_us":67,"threads_started":1}
I20260924 21:11:36.055022 17413 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f5608d721f6d49918c448489f906e37e: Generated new TSK 0
I20260924 21:11:36.055154 17413 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790284076800039-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790284076800039-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260924 21:11:36.084796 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAzNNjS/build/release/bin/kudu
/tmp/dist-test-taskAzNNjS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790284076800039-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790284076800039-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790284076800039-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790284076800039-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34807
--keytab_file=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790284076800039-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45229
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790284076800039-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790284076800039-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790284076800039-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 21:11:36.154708 17427 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 21:11:36.154867 17427 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 21:11:36.154887 17427 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 21:11:36.155953 17427 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 21:11:36.155993 17427 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 21:11:36.156042 17427 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 21:11:36.157531 17427 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45229
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790284076800039-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790284076800039-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790284076800039-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790284076800039-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34807
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17427
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790284076800039-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 19:49:06 UTC on ccycloud.quasar-vhqjoc.root.comops.site
build id 4670864
I20260924 21:11:36.157786 17427 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 21:11:36.158058 17427 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 21:11:36.162381 17435 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 21:11:36.162413 17432 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 21:11:36.162413 17433 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 21:11:36.162658 17427 server_base.cc:1034] running on GCE node
I20260924 21:11:36.162869 17427 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 21:11:36.163074 17427 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 21:11:36.164223 17427 hybrid_clock.cc:648] HybridClock initialized: now 1790284296164216 us; error 35 us; skew 500 ppm
Sep 24 21:11:36 dist-test-slave-c7tk krb5kdc[17320](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284296, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 21:11:36.166414 17427 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 21:11:36.166846 17427 webserver.cc:460] Webserver started at http://127.0.98.1:40831/ using document root <none> and password file <none>
I20260924 21:11:36.167042 17427 fs_manager.cc:359] Metadata directory not provided
I20260924 21:11:36.167097 17427 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 21:11:36.167229 17427 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 21:11:36.168171 17427 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790284076800039-392-0/minicluster-data/ts-0/data/instance:
uuid: "007c54013271403b8c448ae0e3f4ce19"
format_stamp: "Formatted at 2026-09-24 21:11:36 on dist-test-slave-c7tk"
server_key: "4436c88f9b495d5e1d350a19e2958bdb"
server_key_iv: "43183a239376f000dd5ce310104b93f0"
server_key_version: "encryptionkey@0"
I20260924 21:11:36.168510 17427 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790284076800039-392-0/minicluster-data/ts-0/wal/instance:
uuid: "007c54013271403b8c448ae0e3f4ce19"
format_stamp: "Formatted at 2026-09-24 21:11:36 on dist-test-slave-c7tk"
server_key: "4436c88f9b495d5e1d350a19e2958bdb"
server_key_iv: "43183a239376f000dd5ce310104b93f0"
server_key_version: "encryptionkey@0"
I20260924 21:11:36.169691 17427 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260924 21:11:36.170311 17442 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 21:11:36.170466 17427 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 21:11:36.170547 17427 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790284076800039-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790284076800039-392-0/minicluster-data/ts-0/wal
uuid: "007c54013271403b8c448ae0e3f4ce19"
format_stamp: "Formatted at 2026-09-24 21:11:36 on dist-test-slave-c7tk"
server_key: "4436c88f9b495d5e1d350a19e2958bdb"
server_key_iv: "43183a239376f000dd5ce310104b93f0"
server_key_version: "encryptionkey@0"
I20260924 21:11:36.170624 17427 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790284076800039-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790284076800039-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790284076800039-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 21:11:36.181243 17427 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 21:11:36.182708 17427 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 21:11:36.182839 17427 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 21:11:36.183030 17427 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 21:11:36.183276 17427 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 21:11:36.183323 17427 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 21:11:36.183352 17427 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 21:11:36.183375 17427 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 21:11:36.188560 17427 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:42061
I20260924 21:11:36.188601 17555 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:42061 every 8 connection(s)
I20260924 21:11:36.188917 17427 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790284076800039-392-0/minicluster-data/ts-0/data/info.pb
Sep 24 21:11:36 dist-test-slave-c7tk krb5kdc[17320](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284296, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 21:11:36.196346 17424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:11:36.189519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:58257 (local address 127.0.98.62:34807)
0924 21:11:36.189568 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0924 21:11:36.189569 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 21:11:36.189777 (+ 208us) server_negotiation.cc:407] Connection header received
0924 21:11:36.190684 (+ 907us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:11:36.190685 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:11:36.190695 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:11:36.190756 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:11:36.191245 (+ 489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:11:36.191675 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:11:36.192444 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:11:36.192607 (+ 163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:11:36.192649 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:11:36.194249 (+ 1600us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 21:11:36.194250 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 21:11:36.194251 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 21:11:36.194256 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 21:11:36.195427 (+ 1171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:11:36.195757 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:11:36.195758 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:11:36.195758 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:11:36.195798 (+ 40us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:11:36.195940 (+ 142us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:11:36.195941 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:11:36.195941 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:11:36.196019 (+ 78us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 21:11:36.196059 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0924 21:11:36.196241 (+ 182us) server_negotiation.cc:299] Negotiation successful
0924 21:11:36.196294 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260924 21:11:36.196465 17558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:11:36.189501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34807 (local address 127.0.98.1:58257)
0924 21:11:36.189646 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0924 21:11:36.189699 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0924 21:11:36.190596 (+ 897us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 21:11:36.190804 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 21:11:36.190807 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 21:11:36.191030 (+ 223us) client_negotiation.cc:189] Negotiated authn=SASL
0924 21:11:36.191183 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:11:36.191186 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:11:36.191740 (+ 554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:11:36.191740 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:11:36.192383 (+ 643us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:11:36.192384 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:11:36.192667 (+ 283us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:11:36.192668 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:11:36.192758 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:11:36.193097 (+ 339us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 21:11:36.193103 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0924 21:11:36.194177 (+ 1074us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 21:11:36.195484 (+ 1307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 21:11:36.195488 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 21:11:36.195489 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 21:11:36.195702 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 21:11:36.195831 (+ 129us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 21:11:36.195832 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 21:11:36.195832 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 21:11:36.195900 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 21:11:36.196065 (+ 165us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 21:11:36.196069 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 21:11:36.196152 (+ 83us) client_negotiation.cc:769] Sending connection context
0924 21:11:36.196217 (+ 65us) client_negotiation.cc:240] Negotiation successful
0924 21:11:36.196306 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":87,"threads_started":1}
I20260924 21:11:36.196874 17556 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34807
I20260924 21:11:36.196949 17556 heartbeater.cc:464] Registering TS with master...
I20260924 21:11:36.197052 17556 heartbeater.cc:511] Master 127.0.98.62:34807 requested a full tablet report, sending...
I20260924 21:11:36.197460 17365 ts_manager.cc:194] Registered new tserver with Master: 007c54013271403b8c448ae0e3f4ce19 (127.0.98.1:42061)
I20260924 21:11:36.198071 17365 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:58257
I20260924 21:11:36.198537 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAzNNjS/build/release/bin/kudu as pid 17427
I20260924 21:11:36.198640 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790284076800039-392-0/minicluster-data/ts-0/wal/instance
I20260924 21:11:36.198812 392 external_mini_cluster.cc:1428] Setting key 6e1ce2a5b1637774371f2033c8bfa1f1
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790284076800039-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790284076800039-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260924 21:11:36.219749 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAzNNjS/build/release/bin/kudu
/tmp/dist-test-taskAzNNjS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790284076800039-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790284076800039-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790284076800039-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790284076800039-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34807
--keytab_file=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790284076800039-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45229
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790284076800039-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790284076800039-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790284076800039-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 21:11:36.288856 17561 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 21:11:36.289022 17561 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 21:11:36.289045 17561 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 21:11:36.289994 17561 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 21:11:36.290021 17561 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 21:11:36.290042 17561 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 21:11:36.291185 17561 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45229
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790284076800039-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790284076800039-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790284076800039-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790284076800039-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34807
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17561
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790284076800039-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 19:49:06 UTC on ccycloud.quasar-vhqjoc.root.comops.site
build id 4670864
I20260924 21:11:36.291319 17561 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 21:11:36.291446 17561 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 21:11:36.295444 17566 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 21:11:36.295533 17569 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 21:11:36.295533 17567 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 21:11:36.295681 17561 server_base.cc:1034] running on GCE node
I20260924 21:11:36.295869 17561 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 21:11:36.296063 17561 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 21:11:36.297214 17561 hybrid_clock.cc:648] HybridClock initialized: now 1790284296297205 us; error 21 us; skew 500 ppm
Sep 24 21:11:36 dist-test-slave-c7tk krb5kdc[17320](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284296, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 21:11:36.299465 17561 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 21:11:36.299839 17561 webserver.cc:460] Webserver started at http://127.0.98.2:39291/ using document root <none> and password file <none>
I20260924 21:11:36.300016 17561 fs_manager.cc:359] Metadata directory not provided
I20260924 21:11:36.300071 17561 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 21:11:36.300190 17561 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 21:11:36.301020 17561 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790284076800039-392-0/minicluster-data/ts-1/data/instance:
uuid: "8a1424230edc4229b01c99ce47bf05e6"
format_stamp: "Formatted at 2026-09-24 21:11:36 on dist-test-slave-c7tk"
server_key: "24e5b78b60dfd21abe3e88471762a97f"
server_key_iv: "e67205449e54bd73c6d3290b14419317"
server_key_version: "encryptionkey@0"
I20260924 21:11:36.301313 17561 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790284076800039-392-0/minicluster-data/ts-1/wal/instance:
uuid: "8a1424230edc4229b01c99ce47bf05e6"
format_stamp: "Formatted at 2026-09-24 21:11:36 on dist-test-slave-c7tk"
server_key: "24e5b78b60dfd21abe3e88471762a97f"
server_key_iv: "e67205449e54bd73c6d3290b14419317"
server_key_version: "encryptionkey@0"
I20260924 21:11:36.302439 17561 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 21:11:36.303071 17576 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 21:11:36.303208 17561 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 21:11:36.303290 17561 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790284076800039-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790284076800039-392-0/minicluster-data/ts-1/wal
uuid: "8a1424230edc4229b01c99ce47bf05e6"
format_stamp: "Formatted at 2026-09-24 21:11:36 on dist-test-slave-c7tk"
server_key: "24e5b78b60dfd21abe3e88471762a97f"
server_key_iv: "e67205449e54bd73c6d3290b14419317"
server_key_version: "encryptionkey@0"
I20260924 21:11:36.303357 17561 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790284076800039-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790284076800039-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790284076800039-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 21:11:36.313463 17561 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 21:11:36.314940 17561 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 21:11:36.315074 17561 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 21:11:36.315250 17561 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 21:11:36.315483 17561 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 21:11:36.315528 17561 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 21:11:36.315557 17561 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 21:11:36.315587 17561 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 21:11:36.320672 17561 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:37255
I20260924 21:11:36.320701 17689 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:37255 every 8 connection(s)
I20260924 21:11:36.320936 17561 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790284076800039-392-0/minicluster-data/ts-1/data/info.pb
I20260924 21:11:36.323233 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAzNNjS/build/release/bin/kudu as pid 17561
I20260924 21:11:36.323338 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790284076800039-392-0/minicluster-data/ts-1/wal/instance
I20260924 21:11:36.323505 392 external_mini_cluster.cc:1428] Setting key 0ecf9da14af5f8309414a26d3d488355
Sep 24 21:11:36 dist-test-slave-c7tk krb5kdc[17320](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284296, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 21:11:36.330546 17692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:11:36.321495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34807 (local address 127.0.98.2:49007)
0924 21:11:36.321674 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0924 21:11:36.321702 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0924 21:11:36.322557 (+ 855us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 21:11:36.322804 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 21:11:36.322808 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 21:11:36.323042 (+ 234us) client_negotiation.cc:189] Negotiated authn=SASL
0924 21:11:36.323222 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:11:36.323226 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:11:36.323968 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:11:36.323969 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:11:36.325222 (+ 1253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:11:36.325225 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:11:36.325657 (+ 432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:11:36.325658 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:11:36.325754 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:11:36.326277 (+ 523us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 21:11:36.326285 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 21:11:36.327541 (+ 1256us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 21:11:36.329341 (+ 1800us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 21:11:36.329346 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 21:11:36.329347 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 21:11:36.329537 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 21:11:36.329741 (+ 204us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 21:11:36.329742 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 21:11:36.329742 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 21:11:36.329911 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 21:11:36.330166 (+ 255us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 21:11:36.330170 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 21:11:36.330261 (+ 91us) client_negotiation.cc:769] Sending connection context
0924 21:11:36.330326 (+ 65us) client_negotiation.cc:240] Negotiation successful
0924 21:11:36.330419 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":76,"threads_started":1}
I20260924 21:11:36.330559 17424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:11:36.321507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:49007 (local address 127.0.98.62:34807)
0924 21:11:36.321555 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0924 21:11:36.321557 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 21:11:36.321806 (+ 249us) server_negotiation.cc:407] Connection header received
0924 21:11:36.322667 (+ 861us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:11:36.322669 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:11:36.322681 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:11:36.322729 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:11:36.323316 (+ 587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:11:36.323888 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:11:36.325309 (+ 1421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:11:36.325558 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:11:36.325640 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:11:36.327623 (+ 1983us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 21:11:36.327624 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 21:11:36.327625 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 21:11:36.327633 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 21:11:36.329265 (+ 1632us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:11:36.329608 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:11:36.329610 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:11:36.329610 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:11:36.329670 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:11:36.329968 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:11:36.329968 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:11:36.329969 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:11:36.330102 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 21:11:36.330172 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0924 21:11:36.330388 (+ 216us) server_negotiation.cc:299] Negotiation successful
0924 21:11:36.330461 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260924 21:11:36.330932 17690 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34807
I20260924 21:11:36.331037 17690 heartbeater.cc:464] Registering TS with master...
I20260924 21:11:36.331187 17690 heartbeater.cc:511] Master 127.0.98.62:34807 requested a full tablet report, sending...
I20260924 21:11:36.331527 17365 ts_manager.cc:194] Registered new tserver with Master: 8a1424230edc4229b01c99ce47bf05e6 (127.0.98.2:37255)
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260924 21:11:36.332010 17365 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:49007
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790284076800039-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790284076800039-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260924 21:11:36.346477 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAzNNjS/build/release/bin/kudu
/tmp/dist-test-taskAzNNjS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790284076800039-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790284076800039-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790284076800039-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790284076800039-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34807
--keytab_file=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790284076800039-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45229
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790284076800039-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790284076800039-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790284076800039-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 21:11:36.416384 17695 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 21:11:36.416538 17695 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 21:11:36.416558 17695 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 21:11:36.417485 17695 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 21:11:36.417513 17695 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 21:11:36.417536 17695 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 21:11:36.418936 17695 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45229
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790284076800039-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790284076800039-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790284076800039-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790284076800039-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34807
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17695
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790284076800039-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 19:49:06 UTC on ccycloud.quasar-vhqjoc.root.comops.site
build id 4670864
I20260924 21:11:36.419106 17695 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 21:11:36.419253 17695 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 21:11:36.423451 17703 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 21:11:36.423470 17701 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 21:11:36.423475 17700 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 21:11:36.423504 17695 server_base.cc:1034] running on GCE node
I20260924 21:11:36.423780 17695 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 21:11:36.423985 17695 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 21:11:36.425112 17695 hybrid_clock.cc:648] HybridClock initialized: now 1790284296425088 us; error 37 us; skew 500 ppm
Sep 24 21:11:36 dist-test-slave-c7tk krb5kdc[17320](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284296, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 21:11:36.427258 17695 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 21:11:36.427606 17695 webserver.cc:460] Webserver started at http://127.0.98.3:40187/ using document root <none> and password file <none>
I20260924 21:11:36.427795 17695 fs_manager.cc:359] Metadata directory not provided
I20260924 21:11:36.427834 17695 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 21:11:36.427975 17695 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 21:11:36.428885 17695 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790284076800039-392-0/minicluster-data/ts-2/data/instance:
uuid: "adfa31a38655411381ccb92dd594fcd1"
format_stamp: "Formatted at 2026-09-24 21:11:36 on dist-test-slave-c7tk"
server_key: "aad838597e80ecdd09886f21f43c1342"
server_key_iv: "e36512626999a339d5bed6ec86299ef4"
server_key_version: "encryptionkey@0"
I20260924 21:11:36.429248 17695 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790284076800039-392-0/minicluster-data/ts-2/wal/instance:
uuid: "adfa31a38655411381ccb92dd594fcd1"
format_stamp: "Formatted at 2026-09-24 21:11:36 on dist-test-slave-c7tk"
server_key: "aad838597e80ecdd09886f21f43c1342"
server_key_iv: "e36512626999a339d5bed6ec86299ef4"
server_key_version: "encryptionkey@0"
I20260924 21:11:36.430379 17695 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 21:11:36.430943 17710 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 21:11:36.431090 17695 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 21:11:36.431146 17695 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790284076800039-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790284076800039-392-0/minicluster-data/ts-2/wal
uuid: "adfa31a38655411381ccb92dd594fcd1"
format_stamp: "Formatted at 2026-09-24 21:11:36 on dist-test-slave-c7tk"
server_key: "aad838597e80ecdd09886f21f43c1342"
server_key_iv: "e36512626999a339d5bed6ec86299ef4"
server_key_version: "encryptionkey@0"
I20260924 21:11:36.431241 17695 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790284076800039-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790284076800039-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790284076800039-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 21:11:36.438627 17695 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 21:11:36.440507 17695 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 21:11:36.440642 17695 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 21:11:36.440811 17695 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 21:11:36.441053 17695 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 21:11:36.441102 17695 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 21:11:36.441134 17695 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 21:11:36.441157 17695 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 21:11:36.446151 17695 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:46483
I20260924 21:11:36.446211 17823 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:46483 every 8 connection(s)
I20260924 21:11:36.446480 17695 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790284076800039-392-0/minicluster-data/ts-2/data/info.pb
I20260924 21:11:36.450058 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAzNNjS/build/release/bin/kudu as pid 17695
I20260924 21:11:36.450163 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790284076800039-392-0/minicluster-data/ts-2/wal/instance
I20260924 21:11:36.450340 392 external_mini_cluster.cc:1428] Setting key 80f2127354aac6f723a2450bde163968
Sep 24 21:11:36 dist-test-slave-c7tk krb5kdc[17320](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284296, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 21:11:36.454062 17424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:11:36.447138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:60725 (local address 127.0.98.62:34807)
0924 21:11:36.447176 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0924 21:11:36.447177 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 21:11:36.447190 (+ 13us) server_negotiation.cc:407] Connection header received
0924 21:11:36.447888 (+ 698us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:11:36.447889 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:11:36.447897 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:11:36.447970 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:11:36.448456 (+ 486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:11:36.448927 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:11:36.449797 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:11:36.450029 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:11:36.450100 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:11:36.452049 (+ 1949us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 21:11:36.452050 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 21:11:36.452050 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 21:11:36.452056 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 21:11:36.453063 (+ 1007us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:11:36.453367 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:11:36.453367 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:11:36.453367 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:11:36.453409 (+ 42us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:11:36.453571 (+ 162us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:11:36.453572 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:11:36.453572 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:11:36.453652 (+ 80us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 21:11:36.453696 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0924 21:11:36.453946 (+ 250us) server_negotiation.cc:299] Negotiation successful
0924 21:11:36.454006 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260924 21:11:36.454185 17826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:11:36.446989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34807 (local address 127.0.98.3:60725)
0924 21:11:36.447108 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0924 21:11:36.447140 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0924 21:11:36.447811 (+ 671us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 21:11:36.448016 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 21:11:36.448019 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 21:11:36.448232 (+ 213us) client_negotiation.cc:189] Negotiated authn=SASL
0924 21:11:36.448390 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:11:36.448394 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:11:36.448999 (+ 605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:11:36.449000 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:11:36.449720 (+ 720us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:11:36.449722 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:11:36.450106 (+ 384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:11:36.450107 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:11:36.450244 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:11:36.450710 (+ 466us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 21:11:36.450719 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 21:11:36.451983 (+ 1264us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 21:11:36.453125 (+ 1142us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 21:11:36.453130 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 21:11:36.453131 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 21:11:36.453310 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 21:11:36.453444 (+ 134us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 21:11:36.453444 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 21:11:36.453445 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 21:11:36.453526 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 21:11:36.453700 (+ 174us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 21:11:36.453702 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 21:11:36.453789 (+ 87us) client_negotiation.cc:769] Sending connection context
0924 21:11:36.453930 (+ 141us) client_negotiation.cc:240] Negotiation successful
0924 21:11:36.454034 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":90,"thread_start_us":67,"threads_started":1}
I20260924 21:11:36.454623 17824 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34807
I20260924 21:11:36.454705 17824 heartbeater.cc:464] Registering TS with master...
I20260924 21:11:36.454839 17824 heartbeater.cc:511] Master 127.0.98.62:34807 requested a full tablet report, sending...
I20260924 21:11:36.455143 17365 ts_manager.cc:194] Registered new tserver with Master: adfa31a38655411381ccb92dd594fcd1 (127.0.98.3:46483)
I20260924 21:11:36.455483 17365 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:60725
I20260924 21:11:36.462000 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 21:11:36.467670 17424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:11:36.463164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44548 (local address 127.0.98.62:34807)
0924 21:11:36.463193 (+ 29us) server_negotiation.cc:206] Beginning negotiation
0924 21:11:36.463195 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 21:11:36.463285 (+ 90us) server_negotiation.cc:407] Connection header received
0924 21:11:36.463328 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:11:36.463328 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:11:36.463334 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:11:36.463379 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:11:36.463833 (+ 454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:11:36.464226 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:11:36.464807 (+ 581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:11:36.464946 (+ 139us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:11:36.464985 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:11:36.466032 (+ 1047us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 21:11:36.466032 (+ 0us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 21:11:36.466033 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 21:11:36.466037 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 21:11:36.466985 (+ 948us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:11:36.467229 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:11:36.467230 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:11:36.467230 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:11:36.467268 (+ 38us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:11:36.467366 (+ 98us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:11:36.467367 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:11:36.467367 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:11:36.467460 (+ 93us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 21:11:36.467516 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0924 21:11:36.467583 (+ 67us) server_negotiation.cc:299] Negotiation successful
0924 21:11:36.467626 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14}
I20260924 21:11:36.469136 17365 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44548:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 21:11:36.470610 17365 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 21:11:36.476100 17833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:11:36.472321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:37255 (local address 127.0.0.1:55248)
0924 21:11:36.472544 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0924 21:11:36.472553 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 21:11:36.472610 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 21:11:36.473151 (+ 541us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 21:11:36.473152 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 21:11:36.473154 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 21:11:36.473188 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:11:36.473189 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:11:36.474321 (+ 1132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:11:36.474322 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:11:36.474967 (+ 645us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:11:36.474970 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:11:36.475813 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:11:36.475813 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:11:36.475882 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:11:36.475893 (+ 11us) client_negotiation.cc:769] Sending connection context
0924 21:11:36.475956 (+ 63us) client_negotiation.cc:240] Negotiation successful
0924 21:11:36.476005 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":73,"threads_started":1}
I20260924 21:11:36.476286 17832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:11:36.472317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42061 (local address 127.0.0.1:41274)
0924 21:11:36.472452 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0924 21:11:36.472483 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0924 21:11:36.472537 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 21:11:36.472843 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 21:11:36.472846 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 21:11:36.472847 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 21:11:36.472914 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:11:36.472915 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:11:36.474104 (+ 1189us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:11:36.474105 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:11:36.475288 (+ 1183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:11:36.475290 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:11:36.476080 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:11:36.476081 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:11:36.476134 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:11:36.476138 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 21:11:36.476166 (+ 28us) client_negotiation.cc:240] Negotiation successful
0924 21:11:36.476213 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"mutex_wait_us":3,"thread_start_us":46,"threads_started":1}
I20260924 21:11:36.476114 17835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:11:36.472422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55248 (local address 127.0.98.2:37255)
0924 21:11:36.472980 (+ 558us) server_negotiation.cc:206] Beginning negotiation
0924 21:11:36.472984 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 21:11:36.472992 (+ 8us) server_negotiation.cc:407] Connection header received
0924 21:11:36.473028 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:11:36.473030 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:11:36.473058 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:11:36.473252 (+ 194us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 21:11:36.473325 (+ 73us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:11:36.474235 (+ 910us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:11:36.475063 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:11:36.475732 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:11:36.475800 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:11:36.475844 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0924 21:11:36.475981 (+ 137us) server_negotiation.cc:299] Negotiation successful
0924 21:11:36.476043 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":524,"thread_start_us":70,"threads_started":1}
I20260924 21:11:36.476617 17834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:11:36.472400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41274 (local address 127.0.98.1:42061)
0924 21:11:36.472665 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0924 21:11:36.472669 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 21:11:36.472679 (+ 10us) server_negotiation.cc:407] Connection header received
0924 21:11:36.472711 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:11:36.472713 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:11:36.472725 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:11:36.472831 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 21:11:36.472970 (+ 139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:11:36.474004 (+ 1034us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:11:36.475354 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:11:36.475998 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:11:36.476392 (+ 394us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:11:36.476459 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0924 21:11:36.476486 (+ 27us) server_negotiation.cc:299] Negotiation successful
0924 21:11:36.476532 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":228,"thread_start_us":62,"threads_started":1}
I20260924 21:11:36.476956 17624 tablet_service.cc:1467] Processing CreateTablet for tablet f9c682808726493fb24db1482e4bc46c (DEFAULT_TABLE table=test-table [id=764a942cbd134bdba57aba475236d96d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 21:11:36.477308 17624 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f9c682808726493fb24db1482e4bc46c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 21:11:36.477222 17490 tablet_service.cc:1467] Processing CreateTablet for tablet f9c682808726493fb24db1482e4bc46c (DEFAULT_TABLE table=test-table [id=764a942cbd134bdba57aba475236d96d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 21:11:36.477387 17837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:11:36.472610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36738 (local address 127.0.98.3:46483)
0924 21:11:36.472738 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0924 21:11:36.472741 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 21:11:36.472759 (+ 18us) server_negotiation.cc:407] Connection header received
0924 21:11:36.472826 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:11:36.472827 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:11:36.472840 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:11:36.472961 (+ 121us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 21:11:36.474213 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:11:36.475188 (+ 975us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:11:36.476085 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:11:36.476982 (+ 897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:11:36.477066 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:11:36.477114 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0924 21:11:36.477288 (+ 174us) server_negotiation.cc:299] Negotiation successful
0924 21:11:36.477318 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90,"thread_start_us":63,"threads_started":1}
I20260924 21:11:36.477500 17490 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f9c682808726493fb24db1482e4bc46c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 21:11:36.477577 17836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:11:36.472474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:46483 (local address 127.0.0.1:36738)
0924 21:11:36.472622 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0924 21:11:36.472629 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0924 21:11:36.472663 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 21:11:36.474112 (+ 1449us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 21:11:36.474112 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 21:11:36.474114 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 21:11:36.474147 (+ 33us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:11:36.474149 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:11:36.475273 (+ 1124us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:11:36.475274 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:11:36.476014 (+ 740us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:11:36.476016 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:11:36.477067 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:11:36.477068 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:11:36.477189 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:11:36.477193 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 21:11:36.477258 (+ 65us) client_negotiation.cc:240] Negotiation successful
0924 21:11:36.477297 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":49,"threads_started":1}
I20260924 21:11:36.478137 17758 tablet_service.cc:1467] Processing CreateTablet for tablet f9c682808726493fb24db1482e4bc46c (DEFAULT_TABLE table=test-table [id=764a942cbd134bdba57aba475236d96d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 21:11:36.478400 17758 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f9c682808726493fb24db1482e4bc46c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 21:11:36.480461 17838 tablet_bootstrap.cc:492] T f9c682808726493fb24db1482e4bc46c P 8a1424230edc4229b01c99ce47bf05e6: Bootstrap starting.
I20260924 21:11:36.480803 17840 tablet_bootstrap.cc:492] T f9c682808726493fb24db1482e4bc46c P adfa31a38655411381ccb92dd594fcd1: Bootstrap starting.
I20260924 21:11:36.481195 17838 tablet_bootstrap.cc:654] T f9c682808726493fb24db1482e4bc46c P 8a1424230edc4229b01c99ce47bf05e6: Neither blocks nor log segments found. Creating new log.
I20260924 21:11:36.481402 17840 tablet_bootstrap.cc:654] T f9c682808726493fb24db1482e4bc46c P adfa31a38655411381ccb92dd594fcd1: Neither blocks nor log segments found. Creating new log.
I20260924 21:11:36.481772 17838 log.cc:824] T f9c682808726493fb24db1482e4bc46c P 8a1424230edc4229b01c99ce47bf05e6: Log is configured to *not* fsync() on all Append() calls
I20260924 21:11:36.481943 17839 tablet_bootstrap.cc:492] T f9c682808726493fb24db1482e4bc46c P 007c54013271403b8c448ae0e3f4ce19: Bootstrap starting.
I20260924 21:11:36.481973 17840 log.cc:824] T f9c682808726493fb24db1482e4bc46c P adfa31a38655411381ccb92dd594fcd1: Log is configured to *not* fsync() on all Append() calls
I20260924 21:11:36.482487 17839 tablet_bootstrap.cc:654] T f9c682808726493fb24db1482e4bc46c P 007c54013271403b8c448ae0e3f4ce19: Neither blocks nor log segments found. Creating new log.
I20260924 21:11:36.482509 17838 tablet_bootstrap.cc:492] T f9c682808726493fb24db1482e4bc46c P 8a1424230edc4229b01c99ce47bf05e6: No bootstrap required, opened a new log
I20260924 21:11:36.482576 17838 ts_tablet_manager.cc:1397] T f9c682808726493fb24db1482e4bc46c P 8a1424230edc4229b01c99ce47bf05e6: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 21:11:36.482592 17840 tablet_bootstrap.cc:492] T f9c682808726493fb24db1482e4bc46c P adfa31a38655411381ccb92dd594fcd1: No bootstrap required, opened a new log
I20260924 21:11:36.482647 17840 ts_tablet_manager.cc:1397] T f9c682808726493fb24db1482e4bc46c P adfa31a38655411381ccb92dd594fcd1: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 21:11:36.482955 17839 log.cc:824] T f9c682808726493fb24db1482e4bc46c P 007c54013271403b8c448ae0e3f4ce19: Log is configured to *not* fsync() on all Append() calls
I20260924 21:11:36.483654 17839 tablet_bootstrap.cc:492] T f9c682808726493fb24db1482e4bc46c P 007c54013271403b8c448ae0e3f4ce19: No bootstrap required, opened a new log
I20260924 21:11:36.483714 17839 ts_tablet_manager.cc:1397] T f9c682808726493fb24db1482e4bc46c P 007c54013271403b8c448ae0e3f4ce19: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 21:11:36.484067 17838 raft_consensus.cc:348] T f9c682808726493fb24db1482e4bc46c P 8a1424230edc4229b01c99ce47bf05e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "007c54013271403b8c448ae0e3f4ce19" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42061 } } peers { permanent_uuid: "8a1424230edc4229b01c99ce47bf05e6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37255 } } peers { permanent_uuid: "adfa31a38655411381ccb92dd594fcd1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46483 } }
I20260924 21:11:36.484198 17838 raft_consensus.cc:374] T f9c682808726493fb24db1482e4bc46c P 8a1424230edc4229b01c99ce47bf05e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 21:11:36.484236 17838 raft_consensus.cc:729] T f9c682808726493fb24db1482e4bc46c P 8a1424230edc4229b01c99ce47bf05e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a1424230edc4229b01c99ce47bf05e6, State: Initialized, Role: FOLLOWER
I20260924 21:11:36.484324 17838 consensus_queue.cc:260] T f9c682808726493fb24db1482e4bc46c P 8a1424230edc4229b01c99ce47bf05e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "007c54013271403b8c448ae0e3f4ce19" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42061 } } peers { permanent_uuid: "8a1424230edc4229b01c99ce47bf05e6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37255 } } peers { permanent_uuid: "adfa31a38655411381ccb92dd594fcd1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46483 } }
I20260924 21:11:36.484368 17840 raft_consensus.cc:348] T f9c682808726493fb24db1482e4bc46c P adfa31a38655411381ccb92dd594fcd1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "007c54013271403b8c448ae0e3f4ce19" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42061 } } peers { permanent_uuid: "8a1424230edc4229b01c99ce47bf05e6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37255 } } peers { permanent_uuid: "adfa31a38655411381ccb92dd594fcd1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46483 } }
I20260924 21:11:36.484513 17840 raft_consensus.cc:374] T f9c682808726493fb24db1482e4bc46c P adfa31a38655411381ccb92dd594fcd1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 21:11:36.484543 17838 ts_tablet_manager.cc:1428] T f9c682808726493fb24db1482e4bc46c P 8a1424230edc4229b01c99ce47bf05e6: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 21:11:36.484565 17840 raft_consensus.cc:729] T f9c682808726493fb24db1482e4bc46c P adfa31a38655411381ccb92dd594fcd1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: adfa31a38655411381ccb92dd594fcd1, State: Initialized, Role: FOLLOWER
I20260924 21:11:36.484603 17690 heartbeater.cc:503] Master 127.0.98.62:34807 was elected leader, sending a full tablet report...
I20260924 21:11:36.484673 17840 consensus_queue.cc:260] T f9c682808726493fb24db1482e4bc46c P adfa31a38655411381ccb92dd594fcd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "007c54013271403b8c448ae0e3f4ce19" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42061 } } peers { permanent_uuid: "8a1424230edc4229b01c99ce47bf05e6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37255 } } peers { permanent_uuid: "adfa31a38655411381ccb92dd594fcd1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46483 } }
I20260924 21:11:36.484889 17840 ts_tablet_manager.cc:1428] T f9c682808726493fb24db1482e4bc46c P adfa31a38655411381ccb92dd594fcd1: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 21:11:36.484985 17824 heartbeater.cc:503] Master 127.0.98.62:34807 was elected leader, sending a full tablet report...
I20260924 21:11:36.485464 17839 raft_consensus.cc:348] T f9c682808726493fb24db1482e4bc46c P 007c54013271403b8c448ae0e3f4ce19 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "007c54013271403b8c448ae0e3f4ce19" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42061 } } peers { permanent_uuid: "8a1424230edc4229b01c99ce47bf05e6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37255 } } peers { permanent_uuid: "adfa31a38655411381ccb92dd594fcd1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46483 } }
I20260924 21:11:36.485605 17839 raft_consensus.cc:374] T f9c682808726493fb24db1482e4bc46c P 007c54013271403b8c448ae0e3f4ce19 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 21:11:36.485644 17839 raft_consensus.cc:729] T f9c682808726493fb24db1482e4bc46c P 007c54013271403b8c448ae0e3f4ce19 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 007c54013271403b8c448ae0e3f4ce19, State: Initialized, Role: FOLLOWER
I20260924 21:11:36.485736 17839 consensus_queue.cc:260] T f9c682808726493fb24db1482e4bc46c P 007c54013271403b8c448ae0e3f4ce19 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "007c54013271403b8c448ae0e3f4ce19" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42061 } } peers { permanent_uuid: "8a1424230edc4229b01c99ce47bf05e6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37255 } } peers { permanent_uuid: "adfa31a38655411381ccb92dd594fcd1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46483 } }
I20260924 21:11:36.485963 17839 ts_tablet_manager.cc:1428] T f9c682808726493fb24db1482e4bc46c P 007c54013271403b8c448ae0e3f4ce19: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260924 21:11:36.485982 17556 heartbeater.cc:503] Master 127.0.98.62:34807 was elected leader, sending a full tablet report...
W20260924 21:11:36.571492 17691 tablet.cc:2369] T f9c682808726493fb24db1482e4bc46c P 8a1424230edc4229b01c99ce47bf05e6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 21:11:36.621168 17845 raft_consensus.cc:482] T f9c682808726493fb24db1482e4bc46c P adfa31a38655411381ccb92dd594fcd1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 21:11:36.621263 17845 raft_consensus.cc:504] T f9c682808726493fb24db1482e4bc46c P adfa31a38655411381ccb92dd594fcd1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "007c54013271403b8c448ae0e3f4ce19" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42061 } } peers { permanent_uuid: "8a1424230edc4229b01c99ce47bf05e6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37255 } } peers { permanent_uuid: "adfa31a38655411381ccb92dd594fcd1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46483 } }
I20260924 21:11:36.621551 17845 leader_election.cc:288] T f9c682808726493fb24db1482e4bc46c P adfa31a38655411381ccb92dd594fcd1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 007c54013271403b8c448ae0e3f4ce19 (127.0.98.1:42061), 8a1424230edc4229b01c99ce47bf05e6 (127.0.98.2:37255)
I20260924 21:11:36.625422 17835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:11:36.621880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:38479 (local address 127.0.98.2:37255)
0924 21:11:36.621924 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0924 21:11:36.621926 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 21:11:36.622148 (+ 222us) server_negotiation.cc:407] Connection header received
0924 21:11:36.622176 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:11:36.622177 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:11:36.622186 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:11:36.622236 (+ 50us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 21:11:36.622394 (+ 158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:11:36.623074 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:11:36.624571 (+ 1497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:11:36.625093 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:11:36.625160 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:11:36.625169 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0924 21:11:36.625322 (+ 153us) server_negotiation.cc:299] Negotiation successful
0924 21:11:36.625359 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260924 21:11:36.625422 17834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:11:36.621849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:56749 (local address 127.0.98.1:42061)
0924 21:11:36.621894 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0924 21:11:36.621896 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 21:11:36.621909 (+ 13us) server_negotiation.cc:407] Connection header received
0924 21:11:36.622027 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:11:36.622028 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:11:36.622039 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:11:36.622089 (+ 50us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 21:11:36.622284 (+ 195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:11:36.623053 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:11:36.624570 (+ 1517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:11:36.625090 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:11:36.625160 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:11:36.625169 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0924 21:11:36.625322 (+ 153us) server_negotiation.cc:299] Negotiation successful
0924 21:11:36.625359 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260924 21:11:36.625428 17826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:11:36.621756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42061 (local address 127.0.98.3:56749)
0924 21:11:36.621858 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0924 21:11:36.621872 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 21:11:36.621926 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 21:11:36.622115 (+ 189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 21:11:36.622116 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 21:11:36.622118 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 21:11:36.622190 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:11:36.622192 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:11:36.623176 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:11:36.623177 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:11:36.624480 (+ 1303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:11:36.624483 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:11:36.625168 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:11:36.625169 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:11:36.625239 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:11:36.625241 (+ 2us) client_negotiation.cc:769] Sending connection context
0924 21:11:36.625293 (+ 52us) client_negotiation.cc:240] Negotiation successful
0924 21:11:36.625345 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":80,"mutex_wait_us":88}
I20260924 21:11:36.625432 17847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:11:36.621756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:37255 (local address 127.0.98.3:38479)
0924 21:11:36.622040 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0924 21:11:36.622053 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 21:11:36.622126 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 21:11:36.622257 (+ 131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 21:11:36.622258 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 21:11:36.622260 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 21:11:36.622330 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:11:36.622331 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:11:36.623176 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:11:36.623177 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:11:36.624481 (+ 1304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:11:36.624483 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:11:36.625168 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:11:36.625169 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:11:36.625238 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:11:36.625241 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 21:11:36.625293 (+ 52us) client_negotiation.cc:240] Negotiation successful
0924 21:11:36.625345 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":268,"mutex_wait_us":176,"thread_start_us":93,"threads_started":1}
I20260924 21:11:36.625762 17510 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9c682808726493fb24db1482e4bc46c" candidate_uuid: "adfa31a38655411381ccb92dd594fcd1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "007c54013271403b8c448ae0e3f4ce19" is_pre_election: true
I20260924 21:11:36.625762 17644 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9c682808726493fb24db1482e4bc46c" candidate_uuid: "adfa31a38655411381ccb92dd594fcd1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a1424230edc4229b01c99ce47bf05e6" is_pre_election: true
I20260924 21:11:36.625916 17510 raft_consensus.cc:2454] T f9c682808726493fb24db1482e4bc46c P 007c54013271403b8c448ae0e3f4ce19 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate adfa31a38655411381ccb92dd594fcd1 in term 0.
I20260924 21:11:36.625916 17644 raft_consensus.cc:2454] T f9c682808726493fb24db1482e4bc46c P 8a1424230edc4229b01c99ce47bf05e6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate adfa31a38655411381ccb92dd594fcd1 in term 0.
I20260924 21:11:36.626133 17714 leader_election.cc:302] T f9c682808726493fb24db1482e4bc46c P adfa31a38655411381ccb92dd594fcd1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8a1424230edc4229b01c99ce47bf05e6, adfa31a38655411381ccb92dd594fcd1; no voters:
I20260924 21:11:36.626262 17845 raft_consensus.cc:2781] T f9c682808726493fb24db1482e4bc46c P adfa31a38655411381ccb92dd594fcd1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 21:11:36.626312 17845 raft_consensus.cc:482] T f9c682808726493fb24db1482e4bc46c P adfa31a38655411381ccb92dd594fcd1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 21:11:36.626341 17845 raft_consensus.cc:3037] T f9c682808726493fb24db1482e4bc46c P adfa31a38655411381ccb92dd594fcd1 [term 0 FOLLOWER]: Advancing to term 1
I20260924 21:11:36.626971 17845 raft_consensus.cc:504] T f9c682808726493fb24db1482e4bc46c P adfa31a38655411381ccb92dd594fcd1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "007c54013271403b8c448ae0e3f4ce19" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42061 } } peers { permanent_uuid: "8a1424230edc4229b01c99ce47bf05e6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37255 } } peers { permanent_uuid: "adfa31a38655411381ccb92dd594fcd1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46483 } }
I20260924 21:11:36.627105 17845 leader_election.cc:288] T f9c682808726493fb24db1482e4bc46c P adfa31a38655411381ccb92dd594fcd1 [CANDIDATE]: Term 1 election: Requested vote from peers 007c54013271403b8c448ae0e3f4ce19 (127.0.98.1:42061), 8a1424230edc4229b01c99ce47bf05e6 (127.0.98.2:37255)
I20260924 21:11:36.627228 17644 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9c682808726493fb24db1482e4bc46c" candidate_uuid: "adfa31a38655411381ccb92dd594fcd1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a1424230edc4229b01c99ce47bf05e6"
I20260924 21:11:36.627228 17510 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9c682808726493fb24db1482e4bc46c" candidate_uuid: "adfa31a38655411381ccb92dd594fcd1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "007c54013271403b8c448ae0e3f4ce19"
I20260924 21:11:36.627295 17644 raft_consensus.cc:3037] T f9c682808726493fb24db1482e4bc46c P 8a1424230edc4229b01c99ce47bf05e6 [term 0 FOLLOWER]: Advancing to term 1
I20260924 21:11:36.627295 17510 raft_consensus.cc:3037] T f9c682808726493fb24db1482e4bc46c P 007c54013271403b8c448ae0e3f4ce19 [term 0 FOLLOWER]: Advancing to term 1
I20260924 21:11:36.628144 17510 raft_consensus.cc:2454] T f9c682808726493fb24db1482e4bc46c P 007c54013271403b8c448ae0e3f4ce19 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate adfa31a38655411381ccb92dd594fcd1 in term 1.
I20260924 21:11:36.628153 17644 raft_consensus.cc:2454] T f9c682808726493fb24db1482e4bc46c P 8a1424230edc4229b01c99ce47bf05e6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate adfa31a38655411381ccb92dd594fcd1 in term 1.
I20260924 21:11:36.628301 17714 leader_election.cc:302] T f9c682808726493fb24db1482e4bc46c P adfa31a38655411381ccb92dd594fcd1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8a1424230edc4229b01c99ce47bf05e6, adfa31a38655411381ccb92dd594fcd1; no voters:
I20260924 21:11:36.628413 17845 raft_consensus.cc:2781] T f9c682808726493fb24db1482e4bc46c P adfa31a38655411381ccb92dd594fcd1 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 21:11:36.628574 17845 raft_consensus.cc:686] T f9c682808726493fb24db1482e4bc46c P adfa31a38655411381ccb92dd594fcd1 [term 1 LEADER]: Becoming Leader. State: Replica: adfa31a38655411381ccb92dd594fcd1, State: Running, Role: LEADER
I20260924 21:11:36.628677 17845 consensus_queue.cc:237] T f9c682808726493fb24db1482e4bc46c P adfa31a38655411381ccb92dd594fcd1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "007c54013271403b8c448ae0e3f4ce19" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42061 } } peers { permanent_uuid: "8a1424230edc4229b01c99ce47bf05e6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37255 } } peers { permanent_uuid: "adfa31a38655411381ccb92dd594fcd1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46483 } }
I20260924 21:11:36.629469 17365 catalog_manager.cc:5495] T f9c682808726493fb24db1482e4bc46c P adfa31a38655411381ccb92dd594fcd1 reported cstate change: term changed from 0 to 1, leader changed from <none> to adfa31a38655411381ccb92dd594fcd1 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "adfa31a38655411381ccb92dd594fcd1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "007c54013271403b8c448ae0e3f4ce19" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42061 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8a1424230edc4229b01c99ce47bf05e6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37255 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "adfa31a38655411381ccb92dd594fcd1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46483 } health_report { overall_health: HEALTHY } } }
I20260924 21:11:36.658115 17837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:11:36.655969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36750 (local address 127.0.98.3:46483)
0924 21:11:36.656030 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0924 21:11:36.656032 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 21:11:36.656055 (+ 23us) server_negotiation.cc:407] Connection header received
0924 21:11:36.656084 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:11:36.656084 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:11:36.656098 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:11:36.656175 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 21:11:36.656298 (+ 123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:11:36.656687 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:11:36.657371 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:11:36.657552 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:11:36.657617 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:11:36.657765 (+ 148us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 21:11:36.657869 (+ 104us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 21:11:36.657953 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0924 21:11:36.657987 (+ 34us) server_negotiation.cc:299] Negotiation successful
0924 21:11:36.658009 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260924 21:11:36.659677 17644 raft_consensus.cc:1261] T f9c682808726493fb24db1482e4bc46c P 8a1424230edc4229b01c99ce47bf05e6 [term 1 FOLLOWER]: Refusing update from remote peer adfa31a38655411381ccb92dd594fcd1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 21:11:36.659689 17510 raft_consensus.cc:1261] T f9c682808726493fb24db1482e4bc46c P 007c54013271403b8c448ae0e3f4ce19 [term 1 FOLLOWER]: Refusing update from remote peer adfa31a38655411381ccb92dd594fcd1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 21:11:36.659966 17845 consensus_queue.cc:1035] T f9c682808726493fb24db1482e4bc46c P adfa31a38655411381ccb92dd594fcd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a1424230edc4229b01c99ce47bf05e6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37255 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 21:11:36.660113 17848 consensus_queue.cc:1035] T f9c682808726493fb24db1482e4bc46c P adfa31a38655411381ccb92dd594fcd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "007c54013271403b8c448ae0e3f4ce19" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42061 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 21:11:36.662511 17852 mvcc.cc:205] Tried to move back new op lower bound from 7333004479115800576 to 7333004478991699968. Current Snapshot: MvccSnapshot[applied={T|T < 7333004479115800576 or (T in {7333004479115800576})}]
I20260924 21:11:36.664237 17365 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44548:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 21:11:36.664304 17365 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44548:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 21:11:36.664964 17365 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f5608d721f6d49918c448489f906e37e: Sending DeleteTablet for 3 replicas of tablet f9c682808726493fb24db1482e4bc46c
I20260924 21:11:36.665315 17624 tablet_service.cc:1514] Processing DeleteTablet for tablet f9c682808726493fb24db1482e4bc46c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 21:11:36 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:55248
I20260924 21:11:36.665287 17490 tablet_service.cc:1514] Processing DeleteTablet for tablet f9c682808726493fb24db1482e4bc46c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 21:11:36 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:41274
I20260924 21:11:36.665374 17758 tablet_service.cc:1514] Processing DeleteTablet for tablet f9c682808726493fb24db1482e4bc46c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 21:11:36 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:36738
I20260924 21:11:36.665696 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAzNNjS/build/release/bin/kudu with pid 17427
I20260924 21:11:36.667358 17861 tablet_replica.cc:330] T f9c682808726493fb24db1482e4bc46c P 8a1424230edc4229b01c99ce47bf05e6: stopping tablet replica
I20260924 21:11:36.667462 17861 raft_consensus.cc:2229] T f9c682808726493fb24db1482e4bc46c P 8a1424230edc4229b01c99ce47bf05e6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 21:11:36.667563 17861 raft_consensus.cc:2258] T f9c682808726493fb24db1482e4bc46c P 8a1424230edc4229b01c99ce47bf05e6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 21:11:36.667721 17863 tablet_replica.cc:330] T f9c682808726493fb24db1482e4bc46c P adfa31a38655411381ccb92dd594fcd1: stopping tablet replica
I20260924 21:11:36.667819 17863 raft_consensus.cc:2229] T f9c682808726493fb24db1482e4bc46c P adfa31a38655411381ccb92dd594fcd1 [term 1 LEADER]: Raft consensus shutting down.
I20260924 21:11:36.667937 17863 raft_consensus.cc:2258] T f9c682808726493fb24db1482e4bc46c P adfa31a38655411381ccb92dd594fcd1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 21:11:36.668329 17861 ts_tablet_manager.cc:1905] T f9c682808726493fb24db1482e4bc46c P 8a1424230edc4229b01c99ce47bf05e6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 21:11:36.670113 17863 ts_tablet_manager.cc:1905] T f9c682808726493fb24db1482e4bc46c P adfa31a38655411381ccb92dd594fcd1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 21:11:36.670393 17861 ts_tablet_manager.cc:1918] T f9c682808726493fb24db1482e4bc46c P 8a1424230edc4229b01c99ce47bf05e6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 21:11:36.670465 17861 log.cc:1196] T f9c682808726493fb24db1482e4bc46c P 8a1424230edc4229b01c99ce47bf05e6: Deleting WAL directory at /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790284076800039-392-0/minicluster-data/ts-1/wal/wals/f9c682808726493fb24db1482e4bc46c
I20260924 21:11:36.670735 17861 ts_tablet_manager.cc:1939] T f9c682808726493fb24db1482e4bc46c P 8a1424230edc4229b01c99ce47bf05e6: Deleting consensus metadata
I20260924 21:11:36.671347 17352 catalog_manager.cc:4841] TS 8a1424230edc4229b01c99ce47bf05e6 (127.0.98.2:37255): tablet f9c682808726493fb24db1482e4bc46c (table test-table [id=764a942cbd134bdba57aba475236d96d]) successfully deleted
I20260924 21:11:36.671903 17863 ts_tablet_manager.cc:1918] T f9c682808726493fb24db1482e4bc46c P adfa31a38655411381ccb92dd594fcd1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 21:11:36.671965 17863 log.cc:1196] T f9c682808726493fb24db1482e4bc46c P adfa31a38655411381ccb92dd594fcd1: Deleting WAL directory at /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790284076800039-392-0/minicluster-data/ts-2/wal/wals/f9c682808726493fb24db1482e4bc46c
W20260924 21:11:36.672092 17350 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:42061 (error 108)
I20260924 21:11:36.672215 17863 ts_tablet_manager.cc:1939] T f9c682808726493fb24db1482e4bc46c P adfa31a38655411381ccb92dd594fcd1: Deleting consensus metadata
I20260924 21:11:36.672506 17836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:11:36.672413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42061 (local address 127.0.0.1:41284)
0924 21:11:36.672449 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0924 21:11:36.672479 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:42061: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":17}
I20260924 21:11:36.672650 17352 catalog_manager.cc:4841] TS adfa31a38655411381ccb92dd594fcd1 (127.0.98.3:46483): tablet f9c682808726493fb24db1482e4bc46c (table test-table [id=764a942cbd134bdba57aba475236d96d]) successfully deleted
W20260924 21:11:36.672695 17350 catalog_manager.cc:4584] TS 007c54013271403b8c448ae0e3f4ce19 (127.0.98.1:42061): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet f9c682808726493fb24db1482e4bc46c: Network error: Client connection negotiation failed: client connection to 127.0.98.1:42061: connect: Connection refused (error 111)
I20260924 21:11:36.672977 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAzNNjS/build/release/bin/kudu with pid 17561
I20260924 21:11:36.677232 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAzNNjS/build/release/bin/kudu with pid 17695
I20260924 21:11:36.681770 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAzNNjS/build/release/bin/kudu with pid 17333
2026-09-24T21:11:36Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (1805 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-24T21:11:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T21:11:36Z Disabled control of system clock
I20260924 21:11:36.706843 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAzNNjS/build/release/bin/kudu
/tmp/dist-test-taskAzNNjS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790284076800039-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790284076800039-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790284076800039-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790284076800039-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40883
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:40069
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40883
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 21:11:36.776656 17868 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 21:11:36.776818 17868 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 21:11:36.776834 17868 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 21:11:36.777711 17868 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 21:11:36.777738 17868 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 21:11:36.777750 17868 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 21:11:36.777760 17868 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 21:11:36.777770 17868 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 21:11:36.779062 17868 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40069
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790284076800039-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790284076800039-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40883
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40883
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790284076800039-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17868
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790284076800039-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 19:49:06 UTC on ccycloud.quasar-vhqjoc.root.comops.site
build id 4670864
I20260924 21:11:36.779260 17868 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 21:11:36.779433 17868 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 21:11:36.783612 17873 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 21:11:36.783639 17874 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 21:11:36.783677 17868 server_base.cc:1034] running on GCE node
W20260924 21:11:36.783635 17876 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 21:11:36.783944 17868 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 21:11:36.784183 17868 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 21:11:36.785347 17868 hybrid_clock.cc:648] HybridClock initialized: now 1790284296785308 us; error 35 us; skew 500 ppm
I20260924 21:11:36.786504 17868 webserver.cc:460] Webserver started at http://127.0.98.62:38731/ using document root <none> and password file <none>
I20260924 21:11:36.786736 17868 fs_manager.cc:359] Metadata directory not provided
I20260924 21:11:36.786803 17868 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 21:11:36.786932 17868 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 21:11:36.787751 17868 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790284076800039-392-0/minicluster-data/master-0/data/instance:
uuid: "312fd918d9ff437580935e99cebef6e4"
format_stamp: "Formatted at 2026-09-24 21:11:36 on dist-test-slave-c7tk"
server_key: "0d457b34172decb26a2b34545004dd8a"
server_key_iv: "6043b6a1297ab38053fd63fd5212ebeb"
server_key_version: "encryptionkey@0"
I20260924 21:11:36.788089 17868 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790284076800039-392-0/minicluster-data/master-0/wal/instance:
uuid: "312fd918d9ff437580935e99cebef6e4"
format_stamp: "Formatted at 2026-09-24 21:11:36 on dist-test-slave-c7tk"
server_key: "0d457b34172decb26a2b34545004dd8a"
server_key_iv: "6043b6a1297ab38053fd63fd5212ebeb"
server_key_version: "encryptionkey@0"
I20260924 21:11:36.789314 17868 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 21:11:36.790078 17882 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 21:11:36.790241 17868 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260924 21:11:36.790333 17868 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790284076800039-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790284076800039-392-0/minicluster-data/master-0/wal
uuid: "312fd918d9ff437580935e99cebef6e4"
format_stamp: "Formatted at 2026-09-24 21:11:36 on dist-test-slave-c7tk"
server_key: "0d457b34172decb26a2b34545004dd8a"
server_key_iv: "6043b6a1297ab38053fd63fd5212ebeb"
server_key_version: "encryptionkey@0"
I20260924 21:11:36.790410 17868 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790284076800039-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790284076800039-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790284076800039-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 21:11:36.807183 17868 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 21:11:36.807402 17868 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 21:11:36.807520 17868 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 21:11:36.811055 17868 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40883
I20260924 21:11:36.811097 17934 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40883 every 8 connection(s)
I20260924 21:11:36.811359 17868 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790284076800039-392-0/minicluster-data/master-0/data/info.pb
I20260924 21:11:36.812085 17935 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 21:11:36.813774 17935 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 312fd918d9ff437580935e99cebef6e4: Bootstrap starting.
I20260924 21:11:36.814249 17935 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 312fd918d9ff437580935e99cebef6e4: Neither blocks nor log segments found. Creating new log.
I20260924 21:11:36.814522 17935 log.cc:824] T 00000000000000000000000000000000 P 312fd918d9ff437580935e99cebef6e4: Log is configured to *not* fsync() on all Append() calls
I20260924 21:11:36.815078 17935 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 312fd918d9ff437580935e99cebef6e4: No bootstrap required, opened a new log
I20260924 21:11:36.816351 17935 raft_consensus.cc:348] T 00000000000000000000000000000000 P 312fd918d9ff437580935e99cebef6e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "312fd918d9ff437580935e99cebef6e4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40883 } }
I20260924 21:11:36.816458 17935 raft_consensus.cc:374] T 00000000000000000000000000000000 P 312fd918d9ff437580935e99cebef6e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 21:11:36.816476 17935 raft_consensus.cc:729] T 00000000000000000000000000000000 P 312fd918d9ff437580935e99cebef6e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 312fd918d9ff437580935e99cebef6e4, State: Initialized, Role: FOLLOWER
I20260924 21:11:36.816628 17935 consensus_queue.cc:260] T 00000000000000000000000000000000 P 312fd918d9ff437580935e99cebef6e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "312fd918d9ff437580935e99cebef6e4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40883 } }
I20260924 21:11:36.816684 17935 raft_consensus.cc:388] T 00000000000000000000000000000000 P 312fd918d9ff437580935e99cebef6e4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 21:11:36.816704 17935 raft_consensus.cc:482] T 00000000000000000000000000000000 P 312fd918d9ff437580935e99cebef6e4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 21:11:36.816763 17935 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 312fd918d9ff437580935e99cebef6e4 [term 0 FOLLOWER]: Advancing to term 1
I20260924 21:11:36.817168 17935 raft_consensus.cc:504] T 00000000000000000000000000000000 P 312fd918d9ff437580935e99cebef6e4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "312fd918d9ff437580935e99cebef6e4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40883 } }
I20260924 21:11:36.817237 17935 leader_election.cc:302] T 00000000000000000000000000000000 P 312fd918d9ff437580935e99cebef6e4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 312fd918d9ff437580935e99cebef6e4; no voters:
I20260924 21:11:36.817394 17935 leader_election.cc:288] T 00000000000000000000000000000000 P 312fd918d9ff437580935e99cebef6e4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 21:11:36.817452 17938 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 312fd918d9ff437580935e99cebef6e4 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 21:11:36.817601 17938 raft_consensus.cc:686] T 00000000000000000000000000000000 P 312fd918d9ff437580935e99cebef6e4 [term 1 LEADER]: Becoming Leader. State: Replica: 312fd918d9ff437580935e99cebef6e4, State: Running, Role: LEADER
I20260924 21:11:36.817627 17935 sys_catalog.cc:564] T 00000000000000000000000000000000 P 312fd918d9ff437580935e99cebef6e4 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 21:11:36.817688 17938 consensus_queue.cc:237] T 00000000000000000000000000000000 P 312fd918d9ff437580935e99cebef6e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "312fd918d9ff437580935e99cebef6e4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40883 } }
I20260924 21:11:36.818068 17940 sys_catalog.cc:455] T 00000000000000000000000000000000 P 312fd918d9ff437580935e99cebef6e4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 312fd918d9ff437580935e99cebef6e4. Latest consensus state: current_term: 1 leader_uuid: "312fd918d9ff437580935e99cebef6e4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "312fd918d9ff437580935e99cebef6e4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40883 } } }
I20260924 21:11:36.818161 17940 sys_catalog.cc:458] T 00000000000000000000000000000000 P 312fd918d9ff437580935e99cebef6e4 [sys.catalog]: This master's current role is: LEADER
I20260924 21:11:36.818252 17939 sys_catalog.cc:455] T 00000000000000000000000000000000 P 312fd918d9ff437580935e99cebef6e4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "312fd918d9ff437580935e99cebef6e4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "312fd918d9ff437580935e99cebef6e4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40883 } } }
I20260924 21:11:36.818390 17939 sys_catalog.cc:458] T 00000000000000000000000000000000 P 312fd918d9ff437580935e99cebef6e4 [sys.catalog]: This master's current role is: LEADER
I20260924 21:11:36.818498 17947 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 21:11:36.819012 17947 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 21:11:36.820423 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAzNNjS/build/release/bin/kudu as pid 17868
I20260924 21:11:36.820487 17947 catalog_manager.cc:1347] Generated new cluster ID: 947f205a956f45a4842b6c8dd138a422
I20260924 21:11:36.820510 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790284076800039-392-0/minicluster-data/master-0/wal/instance
I20260924 21:11:36.820524 17947 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 21:11:36.820652 392 external_mini_cluster.cc:1428] Setting key 276f511e3d07c69840011e7e7a2ef7a0
I20260924 21:11:36.825850 17958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:11:36.820963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55498 (local address 127.0.98.62:40883)
0924 21:11:36.821988 (+ 1025us) server_negotiation.cc:206] Beginning negotiation
0924 21:11:36.821995 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 21:11:36.822164 (+ 169us) server_negotiation.cc:407] Connection header received
0924 21:11:36.822313 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:11:36.822318 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:11:36.822366 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:11:36.822494 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:11:36.822717 (+ 223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:11:36.823564 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:11:36.824235 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:11:36.824590 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:11:36.824711 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:11:36.825274 (+ 563us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 21:11:36.825277 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 21:11:36.825277 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 21:11:36.825423 (+ 146us) server_negotiation.cc:922] calling sasl_server_start()
0924 21:11:36.825449 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 21:11:36.825457 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 21:11:36.825519 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0924 21:11:36.825586 (+ 67us) server_negotiation.cc:299] Negotiation successful
0924 21:11:36.825650 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":978,"thread_start_us":63,"threads_started":1}
I20260924 21:11:36.851553 17947 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 21:11:36.851960 17947 catalog_manager.cc:1504] Loading token signing keys...
I20260924 21:11:36.861219 17947 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 312fd918d9ff437580935e99cebef6e4: Generated new TSK 0
I20260924 21:11:36.861375 17947 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 21:11:36.869313 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAzNNjS/build/release/bin/kudu
/tmp/dist-test-taskAzNNjS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790284076800039-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790284076800039-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790284076800039-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790284076800039-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40883
--builtin_ntp_servers=127.0.98.20:40069
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 21:11:36.960188 17959 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 21:11:36.960362 17959 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 21:11:36.960381 17959 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 21:11:36.961376 17959 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 21:11:36.961403 17959 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 21:11:36.961441 17959 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 21:11:36.962617 17959 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40069
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790284076800039-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790284076800039-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790284076800039-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40883
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17959
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790284076800039-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 19:49:06 UTC on ccycloud.quasar-vhqjoc.root.comops.site
build id 4670864
I20260924 21:11:36.962757 17959 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 21:11:36.962940 17959 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 21:11:36.966895 17964 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 21:11:36.966907 17967 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 21:11:36.966958 17965 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 21:11:36.967077 17959 server_base.cc:1034] running on GCE node
I20260924 21:11:36.967238 17959 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 21:11:36.967425 17959 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 21:11:36.968573 17959 hybrid_clock.cc:648] HybridClock initialized: now 1790284296968559 us; error 27 us; skew 500 ppm
I20260924 21:11:36.969627 17959 webserver.cc:460] Webserver started at http://127.0.98.1:44043/ using document root <none> and password file <none>
I20260924 21:11:36.969887 17959 fs_manager.cc:359] Metadata directory not provided
I20260924 21:11:36.969928 17959 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 21:11:36.970077 17959 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 21:11:36.970907 17959 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790284076800039-392-0/minicluster-data/ts-0/data/instance:
uuid: "e8e805e5acdc4fbb9d857de95f996a18"
format_stamp: "Formatted at 2026-09-24 21:11:36 on dist-test-slave-c7tk"
server_key: "e808d87bd9c57a71e2bce25b76ebba2e"
server_key_iv: "185aa713bbe1b5c5dfda73dc3f0118ff"
server_key_version: "encryptionkey@0"
I20260924 21:11:36.971206 17959 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790284076800039-392-0/minicluster-data/ts-0/wal/instance:
uuid: "e8e805e5acdc4fbb9d857de95f996a18"
format_stamp: "Formatted at 2026-09-24 21:11:36 on dist-test-slave-c7tk"
server_key: "e808d87bd9c57a71e2bce25b76ebba2e"
server_key_iv: "185aa713bbe1b5c5dfda73dc3f0118ff"
server_key_version: "encryptionkey@0"
I20260924 21:11:36.972249 17959 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 21:11:36.972823 17973 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 21:11:36.972950 17959 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 21:11:36.973075 17959 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790284076800039-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790284076800039-392-0/minicluster-data/ts-0/wal
uuid: "e8e805e5acdc4fbb9d857de95f996a18"
format_stamp: "Formatted at 2026-09-24 21:11:36 on dist-test-slave-c7tk"
server_key: "e808d87bd9c57a71e2bce25b76ebba2e"
server_key_iv: "185aa713bbe1b5c5dfda73dc3f0118ff"
server_key_version: "encryptionkey@0"
I20260924 21:11:36.973153 17959 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790284076800039-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790284076800039-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790284076800039-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 21:11:36.987419 17959 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 21:11:36.987668 17959 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 21:11:36.987789 17959 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 21:11:36.988052 17959 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 21:11:36.988466 17959 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 21:11:36.988500 17959 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 21:11:36.988520 17959 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 21:11:36.988533 17959 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 21:11:36.994068 17959 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:36933
I20260924 21:11:36.994143 18086 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:36933 every 8 connection(s)
I20260924 21:11:36.994391 17959 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790284076800039-392-0/minicluster-data/ts-0/data/info.pb
I20260924 21:11:36.998338 17958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:11:36.994957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:42973 (local address 127.0.98.62:40883)
0924 21:11:36.995031 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0924 21:11:36.995033 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 21:11:36.995189 (+ 156us) server_negotiation.cc:407] Connection header received
0924 21:11:36.995514 (+ 325us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:11:36.995515 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:11:36.995525 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:11:36.995571 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:11:36.996063 (+ 492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:11:36.996462 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:11:36.997223 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:11:36.997373 (+ 150us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:11:36.997416 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:11:36.998089 (+ 673us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 21:11:36.998090 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 21:11:36.998090 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 21:11:36.998094 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 21:11:36.998105 (+ 11us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 21:11:36.998109 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 21:11:36.998148 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0924 21:11:36.998262 (+ 114us) server_negotiation.cc:299] Negotiation successful
0924 21:11:36.998283 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260924 21:11:36.998409 18089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:11:36.994933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40883 (local address 127.0.98.1:42973)
0924 21:11:36.995099 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0924 21:11:36.995123 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0924 21:11:36.995450 (+ 327us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 21:11:36.995612 (+ 162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 21:11:36.995614 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 21:11:36.995621 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0924 21:11:36.995991 (+ 370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:11:36.995994 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:11:36.996527 (+ 533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:11:36.996527 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:11:36.997160 (+ 633us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:11:36.997162 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:11:36.997435 (+ 273us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:11:36.997435 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:11:36.997517 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:11:36.997949 (+ 432us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 21:11:36.997949 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0924 21:11:36.998001 (+ 52us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 21:11:36.998022 (+ 21us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 21:11:36.998148 (+ 126us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 21:11:36.998150 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 21:11:36.998152 (+ 2us) client_negotiation.cc:769] Sending connection context
0924 21:11:36.998196 (+ 44us) client_negotiation.cc:240] Negotiation successful
0924 21:11:36.998259 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":89,"threads_started":1}
I20260924 21:11:36.998865 18087 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40883
I20260924 21:11:36.998957 18087 heartbeater.cc:464] Registering TS with master...
I20260924 21:11:36.999121 18087 heartbeater.cc:511] Master 127.0.98.62:40883 requested a full tablet report, sending...
I20260924 21:11:36.999550 17899 ts_manager.cc:194] Registered new tserver with Master: e8e805e5acdc4fbb9d857de95f996a18 (127.0.98.1:36933)
I20260924 21:11:37.000072 17899 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:42973
I20260924 21:11:37.003938 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAzNNjS/build/release/bin/kudu as pid 17959
I20260924 21:11:37.004086 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790284076800039-392-0/minicluster-data/ts-0/wal/instance
I20260924 21:11:37.004303 392 external_mini_cluster.cc:1428] Setting key c222f251f3ef505bc896c8715cc19004
I20260924 21:11:37.005126 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAzNNjS/build/release/bin/kudu
/tmp/dist-test-taskAzNNjS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790284076800039-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790284076800039-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790284076800039-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790284076800039-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40883
--builtin_ntp_servers=127.0.98.20:40069
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 21:11:37.074828 18090 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 21:11:37.075001 18090 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 21:11:37.075029 18090 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 21:11:37.076077 18090 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 21:11:37.076131 18090 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 21:11:37.076169 18090 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 21:11:37.077541 18090 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40069
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790284076800039-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790284076800039-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790284076800039-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40883
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18090
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790284076800039-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 19:49:06 UTC on ccycloud.quasar-vhqjoc.root.comops.site
build id 4670864
I20260924 21:11:37.077760 18090 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 21:11:37.078081 18090 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 21:11:37.082350 18098 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 21:11:37.082338 18095 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 21:11:37.082406 18090 server_base.cc:1034] running on GCE node
W20260924 21:11:37.082324 18096 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 21:11:37.082747 18090 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 21:11:37.082896 18090 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 21:11:37.084017 18090 hybrid_clock.cc:648] HybridClock initialized: now 1790284297084004 us; error 28 us; skew 500 ppm
I20260924 21:11:37.085016 18090 webserver.cc:460] Webserver started at http://127.0.98.2:42439/ using document root <none> and password file <none>
I20260924 21:11:37.085242 18090 fs_manager.cc:359] Metadata directory not provided
I20260924 21:11:37.085284 18090 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 21:11:37.085417 18090 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 21:11:37.086331 18090 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790284076800039-392-0/minicluster-data/ts-1/data/instance:
uuid: "a7329236bfcd417986ea46d8fabc45d2"
format_stamp: "Formatted at 2026-09-24 21:11:37 on dist-test-slave-c7tk"
server_key: "fc19866495de8c49d7e376039acfda6b"
server_key_iv: "6b719b247e51830594b32210d7e2587c"
server_key_version: "encryptionkey@0"
I20260924 21:11:37.086616 18090 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790284076800039-392-0/minicluster-data/ts-1/wal/instance:
uuid: "a7329236bfcd417986ea46d8fabc45d2"
format_stamp: "Formatted at 2026-09-24 21:11:37 on dist-test-slave-c7tk"
server_key: "fc19866495de8c49d7e376039acfda6b"
server_key_iv: "6b719b247e51830594b32210d7e2587c"
server_key_version: "encryptionkey@0"
I20260924 21:11:37.087644 18090 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 21:11:37.088301 18104 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 21:11:37.088449 18090 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 21:11:37.088552 18090 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790284076800039-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790284076800039-392-0/minicluster-data/ts-1/wal
uuid: "a7329236bfcd417986ea46d8fabc45d2"
format_stamp: "Formatted at 2026-09-24 21:11:37 on dist-test-slave-c7tk"
server_key: "fc19866495de8c49d7e376039acfda6b"
server_key_iv: "6b719b247e51830594b32210d7e2587c"
server_key_version: "encryptionkey@0"
I20260924 21:11:37.088620 18090 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790284076800039-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790284076800039-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790284076800039-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 21:11:37.100857 18090 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 21:11:37.101074 18090 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 21:11:37.101203 18090 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 21:11:37.101375 18090 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 21:11:37.101635 18090 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 21:11:37.101713 18090 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 21:11:37.101747 18090 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 21:11:37.101773 18090 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 21:11:37.106993 18090 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:38493
I20260924 21:11:37.107055 18217 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:38493 every 8 connection(s)
I20260924 21:11:37.107280 18090 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790284076800039-392-0/minicluster-data/ts-1/data/info.pb
I20260924 21:11:37.109025 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAzNNjS/build/release/bin/kudu as pid 18090
I20260924 21:11:37.109126 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790284076800039-392-0/minicluster-data/ts-1/wal/instance
I20260924 21:11:37.109323 392 external_mini_cluster.cc:1428] Setting key d633ac4ebff4a663fdc95c29b0e5f041
I20260924 21:11:37.110124 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAzNNjS/build/release/bin/kudu
/tmp/dist-test-taskAzNNjS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790284076800039-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790284076800039-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790284076800039-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790284076800039-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40883
--builtin_ntp_servers=127.0.98.20:40069
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260924 21:11:37.111848 17958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:11:37.107804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:35309 (local address 127.0.98.62:40883)
0924 21:11:37.107856 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0924 21:11:37.107857 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 21:11:37.108030 (+ 173us) server_negotiation.cc:407] Connection header received
0924 21:11:37.108433 (+ 403us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:11:37.108434 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:11:37.108442 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:11:37.108496 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:11:37.109249 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:11:37.109801 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:11:37.110722 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:11:37.110911 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:11:37.110977 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:11:37.111613 (+ 636us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 21:11:37.111615 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 21:11:37.111615 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 21:11:37.111619 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 21:11:37.111632 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 21:11:37.111635 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 21:11:37.111681 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0924 21:11:37.111759 (+ 78us) server_negotiation.cc:299] Negotiation successful
0924 21:11:37.111783 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260924 21:11:37.112016 18220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:11:37.107764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40883 (local address 127.0.98.2:35309)
0924 21:11:37.107906 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0924 21:11:37.107935 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0924 21:11:37.108352 (+ 417us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 21:11:37.108558 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 21:11:37.108561 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 21:11:37.108568 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0924 21:11:37.109140 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:11:37.109146 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:11:37.109896 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:11:37.109897 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:11:37.110643 (+ 746us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:11:37.110645 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:11:37.110983 (+ 338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:11:37.110983 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:11:37.111105 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:11:37.111495 (+ 390us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 21:11:37.111496 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0924 21:11:37.111516 (+ 20us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 21:11:37.111534 (+ 18us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 21:11:37.111683 (+ 149us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 21:11:37.111687 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 21:11:37.111690 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 21:11:37.111760 (+ 70us) client_negotiation.cc:240] Negotiation successful
0924 21:11:37.111841 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":94,"thread_start_us":72,"threads_started":1}
I20260924 21:11:37.112389 18218 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40883
I20260924 21:11:37.112471 18218 heartbeater.cc:464] Registering TS with master...
I20260924 21:11:37.112628 18218 heartbeater.cc:511] Master 127.0.98.62:40883 requested a full tablet report, sending...
I20260924 21:11:37.112948 17899 ts_manager.cc:194] Registered new tserver with Master: a7329236bfcd417986ea46d8fabc45d2 (127.0.98.2:38493)
I20260924 21:11:37.113282 17899 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:35309
W20260924 21:11:37.179136 18221 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 21:11:37.179311 18221 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 21:11:37.179329 18221 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 21:11:37.180263 18221 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 21:11:37.180296 18221 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 21:11:37.180317 18221 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 21:11:37.181421 18221 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40069
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790284076800039-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790284076800039-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790284076800039-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40883
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18221
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790284076800039-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 19:49:06 UTC on ccycloud.quasar-vhqjoc.root.comops.site
build id 4670864
I20260924 21:11:37.181609 18221 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 21:11:37.181857 18221 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 21:11:37.185858 18227 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 21:11:37.185894 18226 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 21:11:37.185905 18229 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 21:11:37.186020 18221 server_base.cc:1034] running on GCE node
I20260924 21:11:37.186183 18221 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 21:11:37.186403 18221 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 21:11:37.187561 18221 hybrid_clock.cc:648] HybridClock initialized: now 1790284297187537 us; error 32 us; skew 500 ppm
I20260924 21:11:37.188628 18221 webserver.cc:460] Webserver started at http://127.0.98.3:42541/ using document root <none> and password file <none>
I20260924 21:11:37.188858 18221 fs_manager.cc:359] Metadata directory not provided
I20260924 21:11:37.188925 18221 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 21:11:37.189052 18221 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 21:11:37.190133 18221 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790284076800039-392-0/minicluster-data/ts-2/data/instance:
uuid: "cc35eeb1006b4090accb19b39b59a417"
format_stamp: "Formatted at 2026-09-24 21:11:37 on dist-test-slave-c7tk"
server_key: "a5dd18dedba935ac31fd55f4dec59f93"
server_key_iv: "48f44b49d66f2714b967418e3d49038c"
server_key_version: "encryptionkey@0"
I20260924 21:11:37.190465 18221 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790284076800039-392-0/minicluster-data/ts-2/wal/instance:
uuid: "cc35eeb1006b4090accb19b39b59a417"
format_stamp: "Formatted at 2026-09-24 21:11:37 on dist-test-slave-c7tk"
server_key: "a5dd18dedba935ac31fd55f4dec59f93"
server_key_iv: "48f44b49d66f2714b967418e3d49038c"
server_key_version: "encryptionkey@0"
I20260924 21:11:37.191649 18221 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 21:11:37.192317 18235 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 21:11:37.192447 18221 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 21:11:37.192534 18221 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790284076800039-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790284076800039-392-0/minicluster-data/ts-2/wal
uuid: "cc35eeb1006b4090accb19b39b59a417"
format_stamp: "Formatted at 2026-09-24 21:11:37 on dist-test-slave-c7tk"
server_key: "a5dd18dedba935ac31fd55f4dec59f93"
server_key_iv: "48f44b49d66f2714b967418e3d49038c"
server_key_version: "encryptionkey@0"
I20260924 21:11:37.192605 18221 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790284076800039-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790284076800039-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790284076800039-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 21:11:37.210024 18221 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 21:11:37.210216 18221 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 21:11:37.210309 18221 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 21:11:37.210441 18221 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 21:11:37.210655 18221 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 21:11:37.210681 18221 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 21:11:37.210698 18221 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 21:11:37.210709 18221 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 21:11:37.215459 18221 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:38433
I20260924 21:11:37.215528 18348 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:38433 every 8 connection(s)
I20260924 21:11:37.215770 18221 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790284076800039-392-0/minicluster-data/ts-2/data/info.pb
I20260924 21:11:37.220046 17958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:11:37.216325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:34207 (local address 127.0.98.62:40883)
0924 21:11:37.216380 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0924 21:11:37.216382 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 21:11:37.216528 (+ 146us) server_negotiation.cc:407] Connection header received
0924 21:11:37.216884 (+ 356us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:11:37.216885 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:11:37.216892 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:11:37.216938 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:11:37.217421 (+ 483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:11:37.217959 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:11:37.218974 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:11:37.219154 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:11:37.219219 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:11:37.219885 (+ 666us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 21:11:37.219886 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 21:11:37.219886 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 21:11:37.219889 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0924 21:11:37.219897 (+ 8us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 21:11:37.219900 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 21:11:37.219931 (+ 31us) server_negotiation.cc:1035] Waiting for connection context
0924 21:11:37.219979 (+ 48us) server_negotiation.cc:299] Negotiation successful
0924 21:11:37.219996 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260924 21:11:37.220213 18351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:11:37.216303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40883 (local address 127.0.98.3:34207)
0924 21:11:37.216441 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0924 21:11:37.216472 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0924 21:11:37.216817 (+ 345us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 21:11:37.216977 (+ 160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 21:11:37.216979 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 21:11:37.216985 (+ 6us) client_negotiation.cc:189] Negotiated authn=SASL
0924 21:11:37.217353 (+ 368us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:11:37.217355 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:11:37.218062 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:11:37.218063 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:11:37.218908 (+ 845us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:11:37.218910 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:11:37.219236 (+ 326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:11:37.219237 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:11:37.219323 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:11:37.219777 (+ 454us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 21:11:37.219778 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0924 21:11:37.219799 (+ 21us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 21:11:37.219817 (+ 18us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 21:11:37.219932 (+ 115us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 21:11:37.219933 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 21:11:37.219935 (+ 2us) client_negotiation.cc:769] Sending connection context
0924 21:11:37.219972 (+ 37us) client_negotiation.cc:240] Negotiation successful
0924 21:11:37.220029 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":89,"thread_start_us":68,"threads_started":1}
I20260924 21:11:37.220561 18349 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40883
I20260924 21:11:37.220643 18349 heartbeater.cc:464] Registering TS with master...
I20260924 21:11:37.220798 18349 heartbeater.cc:511] Master 127.0.98.62:40883 requested a full tablet report, sending...
I20260924 21:11:37.221096 17899 ts_manager.cc:194] Registered new tserver with Master: cc35eeb1006b4090accb19b39b59a417 (127.0.98.3:38433)
I20260924 21:11:37.221405 17899 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:34207
I20260924 21:11:37.224314 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAzNNjS/build/release/bin/kudu as pid 18221
I20260924 21:11:37.224378 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790284076800039-392-0/minicluster-data/ts-2/wal/instance
I20260924 21:11:37.224494 392 external_mini_cluster.cc:1428] Setting key 8ff732f4f1831f861bd77fdef4efb5b9
I20260924 21:11:37.225176 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 21:11:37.228421 17958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:11:37.226290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55502 (local address 127.0.98.62:40883)
0924 21:11:37.226342 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0924 21:11:37.226343 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 21:11:37.226436 (+ 93us) server_negotiation.cc:407] Connection header received
0924 21:11:37.226468 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:11:37.226469 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:11:37.226476 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:11:37.226521 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:11:37.226627 (+ 106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:11:37.226974 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:11:37.227539 (+ 565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:11:37.227683 (+ 144us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:11:37.227722 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:11:37.228275 (+ 553us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 21:11:37.228275 (+ 0us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 21:11:37.228276 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 21:11:37.228279 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0924 21:11:37.228287 (+ 8us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 21:11:37.228289 (+ 2us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 21:11:37.228321 (+ 32us) server_negotiation.cc:1035] Waiting for connection context
0924 21:11:37.228360 (+ 39us) server_negotiation.cc:299] Negotiation successful
0924 21:11:37.228375 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260924 21:11:37.229723 17899 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55502:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 21:11:37.231269 17899 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 21:11:37.236594 18359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:11:37.233072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:38433 (local address 127.0.0.1:48078)
0924 21:11:37.233231 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0924 21:11:37.233265 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0924 21:11:37.233340 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 21:11:37.233600 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 21:11:37.233602 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 21:11:37.233604 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 21:11:37.233643 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:11:37.233643 (+ 0us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:11:37.234638 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:11:37.234639 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:11:37.235542 (+ 903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:11:37.235544 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:11:37.236346 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:11:37.236347 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:11:37.236414 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:11:37.236424 (+ 10us) client_negotiation.cc:769] Sending connection context
0924 21:11:37.236489 (+ 65us) client_negotiation.cc:240] Negotiation successful
0924 21:11:37.236527 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":53,"threads_started":1}
I20260924 21:11:37.236598 18361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:11:37.233313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48078 (local address 127.0.98.3:38433)
0924 21:11:37.233452 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0924 21:11:37.233455 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 21:11:37.233464 (+ 9us) server_negotiation.cc:407] Connection header received
0924 21:11:37.233494 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:11:37.233495 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:11:37.233507 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:11:37.233574 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 21:11:37.233716 (+ 142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:11:37.234571 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:11:37.235622 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:11:37.236260 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:11:37.236334 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:11:37.236402 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0924 21:11:37.236503 (+ 101us) server_negotiation.cc:299] Negotiation successful
0924 21:11:37.236531 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112,"thread_start_us":65,"threads_started":1}
I20260924 21:11:37.236987 18358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:11:37.233072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36933 (local address 127.0.0.1:35904)
0924 21:11:37.233367 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0924 21:11:37.233376 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 21:11:37.233427 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 21:11:37.233860 (+ 433us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 21:11:37.233861 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 21:11:37.233862 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 21:11:37.233897 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:11:37.233897 (+ 0us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:11:37.235225 (+ 1328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:11:37.235226 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:11:37.236061 (+ 835us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:11:37.236063 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:11:37.236798 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:11:37.236799 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:11:37.236850 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:11:37.236853 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 21:11:37.236900 (+ 47us) client_negotiation.cc:240] Negotiation successful
0924 21:11:37.236933 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":276,"mutex_wait_us":13,"thread_start_us":70,"threads_started":1}
I20260924 21:11:37.237008 18360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:11:37.233201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35904 (local address 127.0.98.1:36933)
0924 21:11:37.233655 (+ 454us) server_negotiation.cc:206] Beginning negotiation
0924 21:11:37.233658 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 21:11:37.233664 (+ 6us) server_negotiation.cc:407] Connection header received
0924 21:11:37.233724 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:11:37.233726 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:11:37.233741 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:11:37.233839 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 21:11:37.234061 (+ 222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:11:37.235143 (+ 1082us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:11:37.236119 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:11:37.236709 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:11:37.236795 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:11:37.236868 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0924 21:11:37.236913 (+ 45us) server_negotiation.cc:299] Negotiation successful
0924 21:11:37.236942 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":423,"thread_start_us":75,"threads_started":1}
I20260924 21:11:37.237432 18357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:11:37.233098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38493 (local address 127.0.0.1:51668)
0924 21:11:37.233935 (+ 837us) negotiation.cc:107] Waiting for socket to connect
0924 21:11:37.233944 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 21:11:37.233998 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 21:11:37.234217 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 21:11:37.234218 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 21:11:37.234219 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 21:11:37.234252 (+ 33us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:11:37.234253 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:11:37.235395 (+ 1142us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:11:37.235396 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:11:37.236366 (+ 970us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:11:37.236368 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:11:37.237228 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:11:37.237229 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:11:37.237285 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:11:37.237288 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 21:11:37.237339 (+ 51us) client_negotiation.cc:240] Negotiation successful
0924 21:11:37.237372 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":824,"mutex_wait_us":1,"thread_start_us":95,"threads_started":1}
I20260924 21:11:37.237372 18283 tablet_service.cc:1467] Processing CreateTablet for tablet 0c6cca34886a44958ca3bb9b3d7f4586 (DEFAULT_TABLE table=test-table [id=a42b9f417e864dcc933e5a7e8bcb3959]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 21:11:37.237466 18362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:11:37.233334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51668 (local address 127.0.98.2:38493)
0924 21:11:37.233489 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0924 21:11:37.233493 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 21:11:37.234055 (+ 562us) server_negotiation.cc:407] Connection header received
0924 21:11:37.234092 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:11:37.234094 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:11:37.234106 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:11:37.234160 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 21:11:37.234319 (+ 159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:11:37.235323 (+ 1004us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:11:37.236440 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:11:37.237064 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:11:37.237132 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:11:37.237199 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0924 21:11:37.237361 (+ 162us) server_negotiation.cc:299] Negotiation successful
0924 21:11:37.237396 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":98,"threads_started":1}
I20260924 21:11:37.237550 18021 tablet_service.cc:1467] Processing CreateTablet for tablet 0c6cca34886a44958ca3bb9b3d7f4586 (DEFAULT_TABLE table=test-table [id=a42b9f417e864dcc933e5a7e8bcb3959]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 21:11:37.237664 18283 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0c6cca34886a44958ca3bb9b3d7f4586. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 21:11:37.237856 18021 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0c6cca34886a44958ca3bb9b3d7f4586. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 21:11:37.238200 18152 tablet_service.cc:1467] Processing CreateTablet for tablet 0c6cca34886a44958ca3bb9b3d7f4586 (DEFAULT_TABLE table=test-table [id=a42b9f417e864dcc933e5a7e8bcb3959]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 21:11:37.238476 18152 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0c6cca34886a44958ca3bb9b3d7f4586. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 21:11:37.239809 18363 tablet_bootstrap.cc:492] T 0c6cca34886a44958ca3bb9b3d7f4586 P cc35eeb1006b4090accb19b39b59a417: Bootstrap starting.
I20260924 21:11:37.240360 18363 tablet_bootstrap.cc:654] T 0c6cca34886a44958ca3bb9b3d7f4586 P cc35eeb1006b4090accb19b39b59a417: Neither blocks nor log segments found. Creating new log.
I20260924 21:11:37.240608 18363 log.cc:824] T 0c6cca34886a44958ca3bb9b3d7f4586 P cc35eeb1006b4090accb19b39b59a417: Log is configured to *not* fsync() on all Append() calls
I20260924 21:11:37.240876 18364 tablet_bootstrap.cc:492] T 0c6cca34886a44958ca3bb9b3d7f4586 P e8e805e5acdc4fbb9d857de95f996a18: Bootstrap starting.
I20260924 21:11:37.241330 18363 tablet_bootstrap.cc:492] T 0c6cca34886a44958ca3bb9b3d7f4586 P cc35eeb1006b4090accb19b39b59a417: No bootstrap required, opened a new log
I20260924 21:11:37.241333 18366 tablet_bootstrap.cc:492] T 0c6cca34886a44958ca3bb9b3d7f4586 P a7329236bfcd417986ea46d8fabc45d2: Bootstrap starting.
I20260924 21:11:37.241423 18364 tablet_bootstrap.cc:654] T 0c6cca34886a44958ca3bb9b3d7f4586 P e8e805e5acdc4fbb9d857de95f996a18: Neither blocks nor log segments found. Creating new log.
I20260924 21:11:37.241447 18363 ts_tablet_manager.cc:1397] T 0c6cca34886a44958ca3bb9b3d7f4586 P cc35eeb1006b4090accb19b39b59a417: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 21:11:37.241716 18364 log.cc:824] T 0c6cca34886a44958ca3bb9b3d7f4586 P e8e805e5acdc4fbb9d857de95f996a18: Log is configured to *not* fsync() on all Append() calls
I20260924 21:11:37.241899 18366 tablet_bootstrap.cc:654] T 0c6cca34886a44958ca3bb9b3d7f4586 P a7329236bfcd417986ea46d8fabc45d2: Neither blocks nor log segments found. Creating new log.
I20260924 21:11:37.242172 18366 log.cc:824] T 0c6cca34886a44958ca3bb9b3d7f4586 P a7329236bfcd417986ea46d8fabc45d2: Log is configured to *not* fsync() on all Append() calls
I20260924 21:11:37.242374 18364 tablet_bootstrap.cc:492] T 0c6cca34886a44958ca3bb9b3d7f4586 P e8e805e5acdc4fbb9d857de95f996a18: No bootstrap required, opened a new log
I20260924 21:11:37.242437 18364 ts_tablet_manager.cc:1397] T 0c6cca34886a44958ca3bb9b3d7f4586 P e8e805e5acdc4fbb9d857de95f996a18: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 21:11:37.242821 18366 tablet_bootstrap.cc:492] T 0c6cca34886a44958ca3bb9b3d7f4586 P a7329236bfcd417986ea46d8fabc45d2: No bootstrap required, opened a new log
I20260924 21:11:37.242913 18366 ts_tablet_manager.cc:1397] T 0c6cca34886a44958ca3bb9b3d7f4586 P a7329236bfcd417986ea46d8fabc45d2: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260924 21:11:37.243876 18364 raft_consensus.cc:348] T 0c6cca34886a44958ca3bb9b3d7f4586 P e8e805e5acdc4fbb9d857de95f996a18 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc35eeb1006b4090accb19b39b59a417" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38433 } } peers { permanent_uuid: "e8e805e5acdc4fbb9d857de95f996a18" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36933 } } peers { permanent_uuid: "a7329236bfcd417986ea46d8fabc45d2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38493 } }
I20260924 21:11:37.243980 18364 raft_consensus.cc:374] T 0c6cca34886a44958ca3bb9b3d7f4586 P e8e805e5acdc4fbb9d857de95f996a18 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 21:11:37.243885 18363 raft_consensus.cc:348] T 0c6cca34886a44958ca3bb9b3d7f4586 P cc35eeb1006b4090accb19b39b59a417 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc35eeb1006b4090accb19b39b59a417" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38433 } } peers { permanent_uuid: "e8e805e5acdc4fbb9d857de95f996a18" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36933 } } peers { permanent_uuid: "a7329236bfcd417986ea46d8fabc45d2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38493 } }
I20260924 21:11:37.243999 18364 raft_consensus.cc:729] T 0c6cca34886a44958ca3bb9b3d7f4586 P e8e805e5acdc4fbb9d857de95f996a18 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e8e805e5acdc4fbb9d857de95f996a18, State: Initialized, Role: FOLLOWER
I20260924 21:11:37.244017 18363 raft_consensus.cc:374] T 0c6cca34886a44958ca3bb9b3d7f4586 P cc35eeb1006b4090accb19b39b59a417 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 21:11:37.244046 18363 raft_consensus.cc:729] T 0c6cca34886a44958ca3bb9b3d7f4586 P cc35eeb1006b4090accb19b39b59a417 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc35eeb1006b4090accb19b39b59a417, State: Initialized, Role: FOLLOWER
I20260924 21:11:37.244095 18364 consensus_queue.cc:260] T 0c6cca34886a44958ca3bb9b3d7f4586 P e8e805e5acdc4fbb9d857de95f996a18 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc35eeb1006b4090accb19b39b59a417" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38433 } } peers { permanent_uuid: "e8e805e5acdc4fbb9d857de95f996a18" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36933 } } peers { permanent_uuid: "a7329236bfcd417986ea46d8fabc45d2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38493 } }
I20260924 21:11:37.244156 18363 consensus_queue.cc:260] T 0c6cca34886a44958ca3bb9b3d7f4586 P cc35eeb1006b4090accb19b39b59a417 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc35eeb1006b4090accb19b39b59a417" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38433 } } peers { permanent_uuid: "e8e805e5acdc4fbb9d857de95f996a18" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36933 } } peers { permanent_uuid: "a7329236bfcd417986ea46d8fabc45d2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38493 } }
I20260924 21:11:37.244313 18364 ts_tablet_manager.cc:1428] T 0c6cca34886a44958ca3bb9b3d7f4586 P e8e805e5acdc4fbb9d857de95f996a18: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 21:11:37.244364 18363 ts_tablet_manager.cc:1428] T 0c6cca34886a44958ca3bb9b3d7f4586 P cc35eeb1006b4090accb19b39b59a417: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 21:11:37.244423 18087 heartbeater.cc:503] Master 127.0.98.62:40883 was elected leader, sending a full tablet report...
I20260924 21:11:37.244467 18349 heartbeater.cc:503] Master 127.0.98.62:40883 was elected leader, sending a full tablet report...
W20260924 21:11:37.244810 18088 tablet.cc:2369] T 0c6cca34886a44958ca3bb9b3d7f4586 P e8e805e5acdc4fbb9d857de95f996a18: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 21:11:37.245520 18366 raft_consensus.cc:348] T 0c6cca34886a44958ca3bb9b3d7f4586 P a7329236bfcd417986ea46d8fabc45d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc35eeb1006b4090accb19b39b59a417" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38433 } } peers { permanent_uuid: "e8e805e5acdc4fbb9d857de95f996a18" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36933 } } peers { permanent_uuid: "a7329236bfcd417986ea46d8fabc45d2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38493 } }
I20260924 21:11:37.245674 18366 raft_consensus.cc:374] T 0c6cca34886a44958ca3bb9b3d7f4586 P a7329236bfcd417986ea46d8fabc45d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 21:11:37.245715 18366 raft_consensus.cc:729] T 0c6cca34886a44958ca3bb9b3d7f4586 P a7329236bfcd417986ea46d8fabc45d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a7329236bfcd417986ea46d8fabc45d2, State: Initialized, Role: FOLLOWER
I20260924 21:11:37.245832 18366 consensus_queue.cc:260] T 0c6cca34886a44958ca3bb9b3d7f4586 P a7329236bfcd417986ea46d8fabc45d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc35eeb1006b4090accb19b39b59a417" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38433 } } peers { permanent_uuid: "e8e805e5acdc4fbb9d857de95f996a18" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36933 } } peers { permanent_uuid: "a7329236bfcd417986ea46d8fabc45d2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38493 } }
I20260924 21:11:37.246064 18366 ts_tablet_manager.cc:1428] T 0c6cca34886a44958ca3bb9b3d7f4586 P a7329236bfcd417986ea46d8fabc45d2: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260924 21:11:37.246160 18218 heartbeater.cc:503] Master 127.0.98.62:40883 was elected leader, sending a full tablet report...
W20260924 21:11:37.357730 18219 tablet.cc:2369] T 0c6cca34886a44958ca3bb9b3d7f4586 P a7329236bfcd417986ea46d8fabc45d2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 21:11:37.384624 18370 raft_consensus.cc:482] T 0c6cca34886a44958ca3bb9b3d7f4586 P cc35eeb1006b4090accb19b39b59a417 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 21:11:37.384748 18370 raft_consensus.cc:504] T 0c6cca34886a44958ca3bb9b3d7f4586 P cc35eeb1006b4090accb19b39b59a417 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc35eeb1006b4090accb19b39b59a417" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38433 } } peers { permanent_uuid: "e8e805e5acdc4fbb9d857de95f996a18" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36933 } } peers { permanent_uuid: "a7329236bfcd417986ea46d8fabc45d2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38493 } }
I20260924 21:11:37.385108 18370 leader_election.cc:288] T 0c6cca34886a44958ca3bb9b3d7f4586 P cc35eeb1006b4090accb19b39b59a417 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e8e805e5acdc4fbb9d857de95f996a18 (127.0.98.1:36933), a7329236bfcd417986ea46d8fabc45d2 (127.0.98.2:38493)
I20260924 21:11:37.387576 18362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:11:37.385324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:36137 (local address 127.0.98.2:38493)
0924 21:11:37.385373 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0924 21:11:37.385375 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 21:11:37.385472 (+ 97us) server_negotiation.cc:407] Connection header received
0924 21:11:37.385512 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:11:37.385512 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:11:37.385521 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:11:37.385567 (+ 46us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 21:11:37.385717 (+ 150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:11:37.386185 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:11:37.386979 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:11:37.387275 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:11:37.387321 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:11:37.387327 (+ 6us) server_negotiation.cc:1035] Waiting for connection context
0924 21:11:37.387480 (+ 153us) server_negotiation.cc:299] Negotiation successful
0924 21:11:37.387521 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260924 21:11:37.387571 18372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:11:37.385256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38493 (local address 127.0.98.3:36137)
0924 21:11:37.385406 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0924 21:11:37.385415 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 21:11:37.385461 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 21:11:37.385575 (+ 114us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 21:11:37.385576 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 21:11:37.385592 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 21:11:37.385656 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:11:37.385658 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:11:37.386251 (+ 593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:11:37.386251 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:11:37.386873 (+ 622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:11:37.386875 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:11:37.387338 (+ 463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:11:37.387339 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:11:37.387408 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:11:37.387411 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 21:11:37.387457 (+ 46us) client_negotiation.cc:240] Negotiation successful
0924 21:11:37.387482 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":84,"threads_started":1}
I20260924 21:11:37.387636 18351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:11:37.385245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36933 (local address 127.0.98.3:47449)
0924 21:11:37.385293 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0924 21:11:37.385304 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 21:11:37.385340 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 21:11:37.385484 (+ 144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 21:11:37.385484 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 21:11:37.385486 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 21:11:37.385552 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:11:37.385554 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:11:37.386177 (+ 623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:11:37.386178 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:11:37.386865 (+ 687us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:11:37.386867 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:11:37.387401 (+ 534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:11:37.387402 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:11:37.387463 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:11:37.387466 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 21:11:37.387515 (+ 49us) client_negotiation.cc:240] Negotiation successful
0924 21:11:37.387543 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32,"mutex_wait_us":24}
I20260924 21:11:37.387638 18360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:11:37.385296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:47449 (local address 127.0.98.1:36933)
0924 21:11:37.385347 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0924 21:11:37.385349 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 21:11:37.385359 (+ 10us) server_negotiation.cc:407] Connection header received
0924 21:11:37.385400 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:11:37.385401 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:11:37.385412 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:11:37.385464 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 21:11:37.385615 (+ 151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:11:37.386115 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:11:37.386943 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:11:37.387332 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:11:37.387389 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:11:37.387399 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0924 21:11:37.387539 (+ 140us) server_negotiation.cc:299] Negotiation successful
0924 21:11:37.387575 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260924 21:11:37.387874 18172 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c6cca34886a44958ca3bb9b3d7f4586" candidate_uuid: "cc35eeb1006b4090accb19b39b59a417" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a7329236bfcd417986ea46d8fabc45d2" is_pre_election: true
I20260924 21:11:37.387908 18041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c6cca34886a44958ca3bb9b3d7f4586" candidate_uuid: "cc35eeb1006b4090accb19b39b59a417" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e8e805e5acdc4fbb9d857de95f996a18" is_pre_election: true
I20260924 21:11:37.388044 18041 raft_consensus.cc:2454] T 0c6cca34886a44958ca3bb9b3d7f4586 P e8e805e5acdc4fbb9d857de95f996a18 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cc35eeb1006b4090accb19b39b59a417 in term 0.
I20260924 21:11:37.388073 18172 raft_consensus.cc:2454] T 0c6cca34886a44958ca3bb9b3d7f4586 P a7329236bfcd417986ea46d8fabc45d2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cc35eeb1006b4090accb19b39b59a417 in term 0.
I20260924 21:11:37.388252 18238 leader_election.cc:302] T 0c6cca34886a44958ca3bb9b3d7f4586 P cc35eeb1006b4090accb19b39b59a417 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cc35eeb1006b4090accb19b39b59a417, e8e805e5acdc4fbb9d857de95f996a18; no voters:
I20260924 21:11:37.388391 18370 raft_consensus.cc:2781] T 0c6cca34886a44958ca3bb9b3d7f4586 P cc35eeb1006b4090accb19b39b59a417 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 21:11:37.388442 18370 raft_consensus.cc:482] T 0c6cca34886a44958ca3bb9b3d7f4586 P cc35eeb1006b4090accb19b39b59a417 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 21:11:37.388495 18370 raft_consensus.cc:3037] T 0c6cca34886a44958ca3bb9b3d7f4586 P cc35eeb1006b4090accb19b39b59a417 [term 0 FOLLOWER]: Advancing to term 1
I20260924 21:11:37.389137 18370 raft_consensus.cc:504] T 0c6cca34886a44958ca3bb9b3d7f4586 P cc35eeb1006b4090accb19b39b59a417 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc35eeb1006b4090accb19b39b59a417" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38433 } } peers { permanent_uuid: "e8e805e5acdc4fbb9d857de95f996a18" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36933 } } peers { permanent_uuid: "a7329236bfcd417986ea46d8fabc45d2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38493 } }
I20260924 21:11:37.389297 18370 leader_election.cc:288] T 0c6cca34886a44958ca3bb9b3d7f4586 P cc35eeb1006b4090accb19b39b59a417 [CANDIDATE]: Term 1 election: Requested vote from peers e8e805e5acdc4fbb9d857de95f996a18 (127.0.98.1:36933), a7329236bfcd417986ea46d8fabc45d2 (127.0.98.2:38493)
I20260924 21:11:37.389432 18041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c6cca34886a44958ca3bb9b3d7f4586" candidate_uuid: "cc35eeb1006b4090accb19b39b59a417" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e8e805e5acdc4fbb9d857de95f996a18"
I20260924 21:11:37.389457 18172 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c6cca34886a44958ca3bb9b3d7f4586" candidate_uuid: "cc35eeb1006b4090accb19b39b59a417" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a7329236bfcd417986ea46d8fabc45d2"
I20260924 21:11:37.389509 18172 raft_consensus.cc:3037] T 0c6cca34886a44958ca3bb9b3d7f4586 P a7329236bfcd417986ea46d8fabc45d2 [term 0 FOLLOWER]: Advancing to term 1
I20260924 21:11:37.389508 18041 raft_consensus.cc:3037] T 0c6cca34886a44958ca3bb9b3d7f4586 P e8e805e5acdc4fbb9d857de95f996a18 [term 0 FOLLOWER]: Advancing to term 1
I20260924 21:11:37.390442 18172 raft_consensus.cc:2454] T 0c6cca34886a44958ca3bb9b3d7f4586 P a7329236bfcd417986ea46d8fabc45d2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cc35eeb1006b4090accb19b39b59a417 in term 1.
I20260924 21:11:37.390442 18041 raft_consensus.cc:2454] T 0c6cca34886a44958ca3bb9b3d7f4586 P e8e805e5acdc4fbb9d857de95f996a18 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cc35eeb1006b4090accb19b39b59a417 in term 1.
I20260924 21:11:37.390604 18238 leader_election.cc:302] T 0c6cca34886a44958ca3bb9b3d7f4586 P cc35eeb1006b4090accb19b39b59a417 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cc35eeb1006b4090accb19b39b59a417, e8e805e5acdc4fbb9d857de95f996a18; no voters:
I20260924 21:11:37.390712 18370 raft_consensus.cc:2781] T 0c6cca34886a44958ca3bb9b3d7f4586 P cc35eeb1006b4090accb19b39b59a417 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 21:11:37.390841 18370 raft_consensus.cc:686] T 0c6cca34886a44958ca3bb9b3d7f4586 P cc35eeb1006b4090accb19b39b59a417 [term 1 LEADER]: Becoming Leader. State: Replica: cc35eeb1006b4090accb19b39b59a417, State: Running, Role: LEADER
I20260924 21:11:37.390929 18370 consensus_queue.cc:237] T 0c6cca34886a44958ca3bb9b3d7f4586 P cc35eeb1006b4090accb19b39b59a417 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc35eeb1006b4090accb19b39b59a417" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38433 } } peers { permanent_uuid: "e8e805e5acdc4fbb9d857de95f996a18" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36933 } } peers { permanent_uuid: "a7329236bfcd417986ea46d8fabc45d2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38493 } }
I20260924 21:11:37.391781 17898 catalog_manager.cc:5495] T 0c6cca34886a44958ca3bb9b3d7f4586 P cc35eeb1006b4090accb19b39b59a417 reported cstate change: term changed from 0 to 1, leader changed from <none> to cc35eeb1006b4090accb19b39b59a417 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "cc35eeb1006b4090accb19b39b59a417" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc35eeb1006b4090accb19b39b59a417" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38433 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e8e805e5acdc4fbb9d857de95f996a18" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36933 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a7329236bfcd417986ea46d8fabc45d2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38493 } health_report { overall_health: UNKNOWN } } }
I20260924 21:11:37.417583 18361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:11:37.415801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48092 (local address 127.0.98.3:38433)
0924 21:11:37.415847 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0924 21:11:37.415848 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 21:11:37.415854 (+ 6us) server_negotiation.cc:407] Connection header received
0924 21:11:37.415874 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:11:37.415875 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:11:37.415887 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:11:37.415941 (+ 54us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 21:11:37.416029 (+ 88us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:11:37.416372 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:11:37.417071 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:11:37.417222 (+ 151us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:11:37.417264 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:11:37.417357 (+ 93us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 21:11:37.417440 (+ 83us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 21:11:37.417484 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0924 21:11:37.417513 (+ 29us) server_negotiation.cc:299] Negotiation successful
0924 21:11:37.417526 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260924 21:11:37.419296 18172 raft_consensus.cc:1261] T 0c6cca34886a44958ca3bb9b3d7f4586 P a7329236bfcd417986ea46d8fabc45d2 [term 1 FOLLOWER]: Refusing update from remote peer cc35eeb1006b4090accb19b39b59a417: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 21:11:37.419296 18041 raft_consensus.cc:1261] T 0c6cca34886a44958ca3bb9b3d7f4586 P e8e805e5acdc4fbb9d857de95f996a18 [term 1 FOLLOWER]: Refusing update from remote peer cc35eeb1006b4090accb19b39b59a417: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 21:11:37.419570 18373 consensus_queue.cc:1035] T 0c6cca34886a44958ca3bb9b3d7f4586 P cc35eeb1006b4090accb19b39b59a417 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a7329236bfcd417986ea46d8fabc45d2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38493 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 21:11:37.419687 18370 consensus_queue.cc:1035] T 0c6cca34886a44958ca3bb9b3d7f4586 P cc35eeb1006b4090accb19b39b59a417 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e8e805e5acdc4fbb9d857de95f996a18" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 21:11:37.423573 17898 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55502:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 21:11:37.423635 17898 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55502:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 21:11:37.424247 17898 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 312fd918d9ff437580935e99cebef6e4: Sending DeleteTablet for 3 replicas of tablet 0c6cca34886a44958ca3bb9b3d7f4586
I20260924 21:11:37.424526 18152 tablet_service.cc:1514] Processing DeleteTablet for tablet 0c6cca34886a44958ca3bb9b3d7f4586 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 21:11:37 UTC) from {username='slave'} at 127.0.0.1:51668
I20260924 21:11:37.424623 18021 tablet_service.cc:1514] Processing DeleteTablet for tablet 0c6cca34886a44958ca3bb9b3d7f4586 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 21:11:37 UTC) from {username='slave'} at 127.0.0.1:35904
I20260924 21:11:37.424723 18386 tablet_replica.cc:330] T 0c6cca34886a44958ca3bb9b3d7f4586 P a7329236bfcd417986ea46d8fabc45d2: stopping tablet replica
I20260924 21:11:37.424811 18387 tablet_replica.cc:330] T 0c6cca34886a44958ca3bb9b3d7f4586 P e8e805e5acdc4fbb9d857de95f996a18: stopping tablet replica
I20260924 21:11:37.424823 18386 raft_consensus.cc:2229] T 0c6cca34886a44958ca3bb9b3d7f4586 P a7329236bfcd417986ea46d8fabc45d2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 21:11:37.424499 18283 tablet_service.cc:1514] Processing DeleteTablet for tablet 0c6cca34886a44958ca3bb9b3d7f4586 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 21:11:37 UTC) from {username='slave'} at 127.0.0.1:48078
I20260924 21:11:37.424889 18387 raft_consensus.cc:2229] T 0c6cca34886a44958ca3bb9b3d7f4586 P e8e805e5acdc4fbb9d857de95f996a18 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 21:11:37.424962 18386 raft_consensus.cc:2258] T 0c6cca34886a44958ca3bb9b3d7f4586 P a7329236bfcd417986ea46d8fabc45d2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 21:11:37.424983 18387 raft_consensus.cc:2258] T 0c6cca34886a44958ca3bb9b3d7f4586 P e8e805e5acdc4fbb9d857de95f996a18 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 21:11:37.425181 18388 tablet_replica.cc:330] T 0c6cca34886a44958ca3bb9b3d7f4586 P cc35eeb1006b4090accb19b39b59a417: stopping tablet replica
I20260924 21:11:37.425230 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAzNNjS/build/release/bin/kudu with pid 17959
I20260924 21:11:37.425261 18388 raft_consensus.cc:2229] T 0c6cca34886a44958ca3bb9b3d7f4586 P cc35eeb1006b4090accb19b39b59a417 [term 1 LEADER]: Raft consensus shutting down.
I20260924 21:11:37.425619 18388 raft_consensus.cc:2258] T 0c6cca34886a44958ca3bb9b3d7f4586 P cc35eeb1006b4090accb19b39b59a417 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 21:11:37.426915 18388 ts_tablet_manager.cc:1905] T 0c6cca34886a44958ca3bb9b3d7f4586 P cc35eeb1006b4090accb19b39b59a417: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 21:11:37.427708 18386 ts_tablet_manager.cc:1905] T 0c6cca34886a44958ca3bb9b3d7f4586 P a7329236bfcd417986ea46d8fabc45d2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 21:11:37.428894 18388 ts_tablet_manager.cc:1918] T 0c6cca34886a44958ca3bb9b3d7f4586 P cc35eeb1006b4090accb19b39b59a417: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 21:11:37.428967 18388 log.cc:1196] T 0c6cca34886a44958ca3bb9b3d7f4586 P cc35eeb1006b4090accb19b39b59a417: Deleting WAL directory at /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790284076800039-392-0/minicluster-data/ts-2/wal/wals/0c6cca34886a44958ca3bb9b3d7f4586
I20260924 21:11:37.429224 18388 ts_tablet_manager.cc:1939] T 0c6cca34886a44958ca3bb9b3d7f4586 P cc35eeb1006b4090accb19b39b59a417: Deleting consensus metadata
I20260924 21:11:37.429397 18386 ts_tablet_manager.cc:1918] T 0c6cca34886a44958ca3bb9b3d7f4586 P a7329236bfcd417986ea46d8fabc45d2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 21:11:37.429497 18386 log.cc:1196] T 0c6cca34886a44958ca3bb9b3d7f4586 P a7329236bfcd417986ea46d8fabc45d2: Deleting WAL directory at /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790284076800039-392-0/minicluster-data/ts-1/wal/wals/0c6cca34886a44958ca3bb9b3d7f4586
I20260924 21:11:37.429533 17884 catalog_manager.cc:4841] TS cc35eeb1006b4090accb19b39b59a417 (127.0.98.3:38433): tablet 0c6cca34886a44958ca3bb9b3d7f4586 (table test-table [id=a42b9f417e864dcc933e5a7e8bcb3959]) successfully deleted
I20260924 21:11:37.429842 18386 ts_tablet_manager.cc:1939] T 0c6cca34886a44958ca3bb9b3d7f4586 P a7329236bfcd417986ea46d8fabc45d2: Deleting consensus metadata
I20260924 21:11:37.430248 17886 catalog_manager.cc:4841] TS a7329236bfcd417986ea46d8fabc45d2 (127.0.98.2:38493): tablet 0c6cca34886a44958ca3bb9b3d7f4586 (table test-table [id=a42b9f417e864dcc933e5a7e8bcb3959]) successfully deleted
W20260924 21:11:37.430384 17885 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:36933 (error 108)
I20260924 21:11:37.430578 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAzNNjS/build/release/bin/kudu with pid 18090
I20260924 21:11:37.431205 18357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:11:37.430705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36933 (local address 127.0.0.1:35920)
0924 21:11:37.431141 (+ 436us) negotiation.cc:107] Waiting for socket to connect
0924 21:11:37.431177 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:36933: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":418}
W20260924 21:11:37.431350 17885 catalog_manager.cc:4584] TS e8e805e5acdc4fbb9d857de95f996a18 (127.0.98.1:36933): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 0c6cca34886a44958ca3bb9b3d7f4586: Network error: Client connection negotiation failed: client connection to 127.0.98.1:36933: connect: Connection refused (error 111)
I20260924 21:11:37.435407 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAzNNjS/build/release/bin/kudu with pid 18221
I20260924 21:11:37.439409 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAzNNjS/build/release/bin/kudu with pid 17868
2026-09-24T21:11:37Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (756 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790284076800039-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 21:11:37 dist-test-slave-c7tk krb5kdc[18392](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 21:11:37 dist-test-slave-c7tk krb5kdc[18392](info): set up 2 sockets
Sep 24 21:11:37 dist-test-slave-c7tk krb5kdc[18392](info): commencing operation
krb5kdc: starting...
W20260924 21:11:38.373368 392 mini_kdc.cc:121] Time spent starting KDC: real 0.916s user 0.000s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790284076800039-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790284076800039-392-0/krb5kdc/test-user.keytab.
Sep 24 21:11:38 dist-test-slave-c7tk krb5kdc[18392](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284298, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T21:11:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T21:11:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790284076800039-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790284076800039-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 21:11:38.472186 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAzNNjS/build/release/bin/kudu
/tmp/dist-test-taskAzNNjS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790284076800039-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790284076800039-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790284076800039-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790284076800039-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33695
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790284076800039-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39243
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33695
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790284076800039-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790284076800039-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790284076800039-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 21:11:38.545509 18405 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 21:11:38.545686 18405 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 21:11:38.545706 18405 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 21:11:38.546743 18405 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 21:11:38.546782 18405 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 21:11:38.546794 18405 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 21:11:38.546834 18405 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 21:11:38.546859 18405 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 21:11:38.548066 18405 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39243
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790284076800039-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790284076800039-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33695
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790284076800039-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33695
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790284076800039-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18405
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790284076800039-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 19:49:06 UTC on ccycloud.quasar-vhqjoc.root.comops.site
build id 4670864
I20260924 21:11:38.548323 18405 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 21:11:38.548543 18405 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 21:11:38.552891 18405 server_base.cc:1034] running on GCE node
W20260924 21:11:38.552838 18413 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 21:11:38.552850 18410 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 21:11:38.552856 18411 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 21:11:38.553256 18405 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 21:11:38.553550 18405 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 21:11:38.554704 18405 hybrid_clock.cc:648] HybridClock initialized: now 1790284298554691 us; error 25 us; skew 500 ppm
Sep 24 21:11:38 dist-test-slave-c7tk krb5kdc[18392](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284298, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 21:11:38.556970 18405 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 21:11:38.557372 18405 webserver.cc:460] Webserver started at http://127.0.98.62:35771/ using document root <none> and password file <none>
I20260924 21:11:38.557571 18405 fs_manager.cc:359] Metadata directory not provided
I20260924 21:11:38.557641 18405 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 21:11:38.557777 18405 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 21:11:38.558760 18405 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790284076800039-392-0/minicluster-data/master-0/data/instance:
uuid: "dd2bd66449f741dfb7d4214ae641cad9"
format_stamp: "Formatted at 2026-09-24 21:11:38 on dist-test-slave-c7tk"
server_key: "a0055456db1a4255fa89bc41c15793d2"
server_key_iv: "504915eb181212fbae1a08a47dd9d09d"
server_key_version: "encryptionkey@0"
I20260924 21:11:38.559105 18405 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790284076800039-392-0/minicluster-data/master-0/wal/instance:
uuid: "dd2bd66449f741dfb7d4214ae641cad9"
format_stamp: "Formatted at 2026-09-24 21:11:38 on dist-test-slave-c7tk"
server_key: "a0055456db1a4255fa89bc41c15793d2"
server_key_iv: "504915eb181212fbae1a08a47dd9d09d"
server_key_version: "encryptionkey@0"
I20260924 21:11:38.560308 18405 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 21:11:38.560925 18420 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 21:11:38.561082 18405 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 21:11:38.561160 18405 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790284076800039-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790284076800039-392-0/minicluster-data/master-0/wal
uuid: "dd2bd66449f741dfb7d4214ae641cad9"
format_stamp: "Formatted at 2026-09-24 21:11:38 on dist-test-slave-c7tk"
server_key: "a0055456db1a4255fa89bc41c15793d2"
server_key_iv: "504915eb181212fbae1a08a47dd9d09d"
server_key_version: "encryptionkey@0"
I20260924 21:11:38.561228 18405 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790284076800039-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790284076800039-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790284076800039-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 21:11:38.583717 18405 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 21:11:38.585250 18405 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 21:11:38.585388 18405 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 21:11:38.589322 18405 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33695
I20260924 21:11:38.589352 18472 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33695 every 8 connection(s)
I20260924 21:11:38.589682 18405 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790284076800039-392-0/minicluster-data/master-0/data/info.pb
I20260924 21:11:38.590509 18473 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 21:11:38.592417 18473 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dd2bd66449f741dfb7d4214ae641cad9: Bootstrap starting.
I20260924 21:11:38.592983 18473 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dd2bd66449f741dfb7d4214ae641cad9: Neither blocks nor log segments found. Creating new log.
I20260924 21:11:38.593271 18473 log.cc:824] T 00000000000000000000000000000000 P dd2bd66449f741dfb7d4214ae641cad9: Log is configured to *not* fsync() on all Append() calls
I20260924 21:11:38.593942 18473 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dd2bd66449f741dfb7d4214ae641cad9: No bootstrap required, opened a new log
I20260924 21:11:38.595470 18473 raft_consensus.cc:348] T 00000000000000000000000000000000 P dd2bd66449f741dfb7d4214ae641cad9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd2bd66449f741dfb7d4214ae641cad9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33695 } }
I20260924 21:11:38.595595 18473 raft_consensus.cc:374] T 00000000000000000000000000000000 P dd2bd66449f741dfb7d4214ae641cad9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 21:11:38.595618 18473 raft_consensus.cc:729] T 00000000000000000000000000000000 P dd2bd66449f741dfb7d4214ae641cad9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd2bd66449f741dfb7d4214ae641cad9, State: Initialized, Role: FOLLOWER
I20260924 21:11:38.595790 18473 consensus_queue.cc:260] T 00000000000000000000000000000000 P dd2bd66449f741dfb7d4214ae641cad9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd2bd66449f741dfb7d4214ae641cad9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33695 } }
I20260924 21:11:38.595850 18473 raft_consensus.cc:388] T 00000000000000000000000000000000 P dd2bd66449f741dfb7d4214ae641cad9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 21:11:38.595890 18473 raft_consensus.cc:482] T 00000000000000000000000000000000 P dd2bd66449f741dfb7d4214ae641cad9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 21:11:38.595940 18473 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dd2bd66449f741dfb7d4214ae641cad9 [term 0 FOLLOWER]: Advancing to term 1
I20260924 21:11:38.596185 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAzNNjS/build/release/bin/kudu as pid 18405
I20260924 21:11:38.596297 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790284076800039-392-0/minicluster-data/master-0/wal/instance
I20260924 21:11:38.596498 392 external_mini_cluster.cc:1428] Setting key 8a2f7e7cf130687fd0a3966beb7db9f8
I20260924 21:11:38.596576 18473 raft_consensus.cc:504] T 00000000000000000000000000000000 P dd2bd66449f741dfb7d4214ae641cad9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd2bd66449f741dfb7d4214ae641cad9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33695 } }
I20260924 21:11:38.597049 18473 leader_election.cc:302] T 00000000000000000000000000000000 P dd2bd66449f741dfb7d4214ae641cad9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dd2bd66449f741dfb7d4214ae641cad9; no voters:
I20260924 21:11:38.597229 18473 leader_election.cc:288] T 00000000000000000000000000000000 P dd2bd66449f741dfb7d4214ae641cad9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 21:11:38.597256 18478 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dd2bd66449f741dfb7d4214ae641cad9 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 21:11:38.597389 18478 raft_consensus.cc:686] T 00000000000000000000000000000000 P dd2bd66449f741dfb7d4214ae641cad9 [term 1 LEADER]: Becoming Leader. State: Replica: dd2bd66449f741dfb7d4214ae641cad9, State: Running, Role: LEADER
I20260924 21:11:38.597468 18478 consensus_queue.cc:237] T 00000000000000000000000000000000 P dd2bd66449f741dfb7d4214ae641cad9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd2bd66449f741dfb7d4214ae641cad9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33695 } }
I20260924 21:11:38.597504 18473 sys_catalog.cc:564] T 00000000000000000000000000000000 P dd2bd66449f741dfb7d4214ae641cad9 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 21:11:38.597810 18478 sys_catalog.cc:455] T 00000000000000000000000000000000 P dd2bd66449f741dfb7d4214ae641cad9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader dd2bd66449f741dfb7d4214ae641cad9. Latest consensus state: current_term: 1 leader_uuid: "dd2bd66449f741dfb7d4214ae641cad9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd2bd66449f741dfb7d4214ae641cad9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33695 } } }
I20260924 21:11:38.597875 18478 sys_catalog.cc:458] T 00000000000000000000000000000000 P dd2bd66449f741dfb7d4214ae641cad9 [sys.catalog]: This master's current role is: LEADER
I20260924 21:11:38.597965 18479 sys_catalog.cc:455] T 00000000000000000000000000000000 P dd2bd66449f741dfb7d4214ae641cad9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dd2bd66449f741dfb7d4214ae641cad9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd2bd66449f741dfb7d4214ae641cad9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33695 } } }
I20260924 21:11:38.598045 18479 sys_catalog.cc:458] T 00000000000000000000000000000000 P dd2bd66449f741dfb7d4214ae641cad9 [sys.catalog]: This master's current role is: LEADER
I20260924 21:11:38.598115 18482 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 21:11:38.598634 18482 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 21:11:38.600162 18482 catalog_manager.cc:1347] Generated new cluster ID: d04044c6aa2e40028b2b98efb80cba3e
I20260924 21:11:38.600208 18482 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 24 21:11:38 dist-test-slave-c7tk krb5kdc[18392](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284298, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 21:11:38.607712 18477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:11:38.596976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59536 (local address 127.0.98.62:33695)
0924 21:11:38.597199 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0924 21:11:38.597210 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 21:11:38.597239 (+ 29us) server_negotiation.cc:407] Connection header received
0924 21:11:38.597379 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:11:38.597386 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:11:38.597441 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:11:38.598393 (+ 952us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:11:38.598495 (+ 102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:11:38.599866 (+ 1371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:11:38.600824 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:11:38.601382 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:11:38.601486 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:11:38.604072 (+ 2586us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 21:11:38.604075 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 21:11:38.604075 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 21:11:38.604085 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0924 21:11:38.606229 (+ 2144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:11:38.606590 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:11:38.606593 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:11:38.606593 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:11:38.606694 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:11:38.606888 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:11:38.606889 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:11:38.606889 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:11:38.607134 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 21:11:38.607263 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0924 21:11:38.607366 (+ 103us) server_negotiation.cc:299] Negotiation successful
0924 21:11:38.607495 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":179,"thread_start_us":97,"threads_started":1}
I20260924 21:11:38.616585 18482 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 21:11:38.617066 18482 catalog_manager.cc:1504] Loading token signing keys...
I20260924 21:11:38.625010 18482 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dd2bd66449f741dfb7d4214ae641cad9: Generated new TSK 0
I20260924 21:11:38.625223 18482 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790284076800039-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790284076800039-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260924 21:11:38.653389 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAzNNjS/build/release/bin/kudu
/tmp/dist-test-taskAzNNjS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790284076800039-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790284076800039-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790284076800039-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790284076800039-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33695
--keytab_file=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790284076800039-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39243
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790284076800039-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790284076800039-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790284076800039-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 21:11:38.729863 18499 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 21:11:38.730082 18499 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 21:11:38.730116 18499 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 21:11:38.731199 18499 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 21:11:38.731258 18499 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 21:11:38.731298 18499 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 21:11:38.732772 18499 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39243
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790284076800039-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790284076800039-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790284076800039-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790284076800039-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33695
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18499
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790284076800039-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 19:49:06 UTC on ccycloud.quasar-vhqjoc.root.comops.site
build id 4670864
I20260924 21:11:38.733026 18499 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 21:11:38.733242 18499 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 21:11:38.737706 18505 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 21:11:38.737720 18504 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 21:11:38.737708 18507 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 21:11:38.737987 18499 server_base.cc:1034] running on GCE node
I20260924 21:11:38.738271 18499 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 21:11:38.738490 18499 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 21:11:38.739638 18499 hybrid_clock.cc:648] HybridClock initialized: now 1790284298739628 us; error 29 us; skew 500 ppm
Sep 24 21:11:38 dist-test-slave-c7tk krb5kdc[18392](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284298, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 21:11:38.746024 18499 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 21:11:38.746383 18499 webserver.cc:460] Webserver started at http://127.0.98.1:34597/ using document root <none> and password file <none>
I20260924 21:11:38.746573 18499 fs_manager.cc:359] Metadata directory not provided
I20260924 21:11:38.746628 18499 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 21:11:38.746752 18499 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 21:11:38.747659 18499 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790284076800039-392-0/minicluster-data/ts-0/data/instance:
uuid: "6b94a3c5013941e6bddecd22313cbb6c"
format_stamp: "Formatted at 2026-09-24 21:11:38 on dist-test-slave-c7tk"
server_key: "653813e6f6f3443475e02d85d605c830"
server_key_iv: "726faab950ab4d903cc0a764958c5c6a"
server_key_version: "encryptionkey@0"
I20260924 21:11:38.747957 18499 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790284076800039-392-0/minicluster-data/ts-0/wal/instance:
uuid: "6b94a3c5013941e6bddecd22313cbb6c"
format_stamp: "Formatted at 2026-09-24 21:11:38 on dist-test-slave-c7tk"
server_key: "653813e6f6f3443475e02d85d605c830"
server_key_iv: "726faab950ab4d903cc0a764958c5c6a"
server_key_version: "encryptionkey@0"
I20260924 21:11:38.749168 18499 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 21:11:38.749889 18514 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 21:11:38.750048 18499 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 21:11:38.750137 18499 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790284076800039-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790284076800039-392-0/minicluster-data/ts-0/wal
uuid: "6b94a3c5013941e6bddecd22313cbb6c"
format_stamp: "Formatted at 2026-09-24 21:11:38 on dist-test-slave-c7tk"
server_key: "653813e6f6f3443475e02d85d605c830"
server_key_iv: "726faab950ab4d903cc0a764958c5c6a"
server_key_version: "encryptionkey@0"
I20260924 21:11:38.750209 18499 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790284076800039-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790284076800039-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790284076800039-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 21:11:38.779458 18499 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 21:11:38.780906 18499 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 21:11:38.781039 18499 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 21:11:38.781229 18499 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 21:11:38.781474 18499 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 21:11:38.781522 18499 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 21:11:38.781550 18499 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 21:11:38.781574 18499 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 21:11:38.786641 18499 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:35799
I20260924 21:11:38.786681 18627 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:35799 every 8 connection(s)
I20260924 21:11:38.786926 18499 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790284076800039-392-0/minicluster-data/ts-0/data/info.pb
I20260924 21:11:38.787922 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAzNNjS/build/release/bin/kudu as pid 18499
I20260924 21:11:38.788012 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790284076800039-392-0/minicluster-data/ts-0/wal/instance
I20260924 21:11:38.788235 392 external_mini_cluster.cc:1428] Setting key 4f1239ccdcd96e1e5fca07affc2fe21a
Sep 24 21:11:38 dist-test-slave-c7tk krb5kdc[18392](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284298, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260924 21:11:38.796072 18477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:11:38.787490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:37195 (local address 127.0.98.62:33695)
0924 21:11:38.787555 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0924 21:11:38.787557 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 21:11:38.787677 (+ 120us) server_negotiation.cc:407] Connection header received
0924 21:11:38.788816 (+ 1139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:11:38.788817 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:11:38.788830 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:11:38.788884 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:11:38.789731 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:11:38.790340 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:11:38.791298 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:11:38.791488 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:11:38.791550 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:11:38.793495 (+ 1945us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 21:11:38.793497 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 21:11:38.793497 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 21:11:38.793504 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 21:11:38.794807 (+ 1303us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:11:38.795173 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:11:38.795174 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:11:38.795175 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:11:38.795228 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:11:38.795443 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:11:38.795443 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:11:38.795444 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:11:38.795550 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 21:11:38.795637 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0924 21:11:38.795915 (+ 278us) server_negotiation.cc:299] Negotiation successful
0924 21:11:38.795991 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260924 21:11:38.796206 18630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:11:38.787448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33695 (local address 127.0.98.1:37195)
0924 21:11:38.787582 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0924 21:11:38.787609 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0924 21:11:38.788722 (+ 1113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 21:11:38.788975 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 21:11:38.788979 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 21:11:38.789357 (+ 378us) client_negotiation.cc:189] Negotiated authn=SASL
0924 21:11:38.789617 (+ 260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:11:38.789622 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:11:38.790432 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:11:38.790433 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:11:38.791199 (+ 766us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:11:38.791201 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:11:38.791567 (+ 366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:11:38.791568 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:11:38.791677 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:11:38.792072 (+ 395us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 21:11:38.792078 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0924 21:11:38.793405 (+ 1327us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 21:11:38.794895 (+ 1490us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 21:11:38.794902 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 21:11:38.794904 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 21:11:38.795090 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 21:11:38.795290 (+ 200us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 21:11:38.795291 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 21:11:38.795291 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 21:11:38.795380 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 21:11:38.795637 (+ 257us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 21:11:38.795644 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 21:11:38.795791 (+ 147us) client_negotiation.cc:769] Sending connection context
0924 21:11:38.795881 (+ 90us) client_negotiation.cc:240] Negotiation successful
0924 21:11:38.796012 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":84,"threads_started":1}
I20260924 21:11:38.796706 18628 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33695
I20260924 21:11:38.796801 18628 heartbeater.cc:464] Registering TS with master...
I20260924 21:11:38.796952 18628 heartbeater.cc:511] Master 127.0.98.62:33695 requested a full tablet report, sending...
I20260924 21:11:38.797456 18437 ts_manager.cc:194] Registered new tserver with Master: 6b94a3c5013941e6bddecd22313cbb6c (127.0.98.1:35799)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260924 21:11:38.798235 18437 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:37195
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790284076800039-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790284076800039-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260924 21:11:38.811830 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAzNNjS/build/release/bin/kudu
/tmp/dist-test-taskAzNNjS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790284076800039-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790284076800039-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790284076800039-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790284076800039-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33695
--keytab_file=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790284076800039-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39243
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790284076800039-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790284076800039-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790284076800039-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 21:11:38.882243 18633 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 21:11:38.882432 18633 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 21:11:38.882465 18633 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 21:11:38.883507 18633 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 21:11:38.883599 18633 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 21:11:38.883641 18633 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 21:11:38.884999 18633 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39243
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790284076800039-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790284076800039-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790284076800039-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790284076800039-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33695
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18633
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790284076800039-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 19:49:06 UTC on ccycloud.quasar-vhqjoc.root.comops.site
build id 4670864
I20260924 21:11:38.885262 18633 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 21:11:38.885450 18633 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 21:11:38.889932 18639 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 21:11:38.890000 18641 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 21:11:38.889919 18638 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 21:11:38.890156 18633 server_base.cc:1034] running on GCE node
I20260924 21:11:38.890390 18633 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 21:11:38.890622 18633 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 21:11:38.891762 18633 hybrid_clock.cc:648] HybridClock initialized: now 1790284298891756 us; error 30 us; skew 500 ppm
Sep 24 21:11:38 dist-test-slave-c7tk krb5kdc[18392](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284298, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 21:11:38.894395 18633 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 21:11:38.894768 18633 webserver.cc:460] Webserver started at http://127.0.98.2:41155/ using document root <none> and password file <none>
I20260924 21:11:38.894965 18633 fs_manager.cc:359] Metadata directory not provided
I20260924 21:11:38.895023 18633 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 21:11:38.895164 18633 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 21:11:38.896013 18633 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790284076800039-392-0/minicluster-data/ts-1/data/instance:
uuid: "182c45293918451a9820fb5399eb7443"
format_stamp: "Formatted at 2026-09-24 21:11:38 on dist-test-slave-c7tk"
server_key: "fe42203acf4e4edab0d12f4dbcb25617"
server_key_iv: "65b4f264956bd85e85de7378624afd27"
server_key_version: "encryptionkey@0"
I20260924 21:11:38.896317 18633 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790284076800039-392-0/minicluster-data/ts-1/wal/instance:
uuid: "182c45293918451a9820fb5399eb7443"
format_stamp: "Formatted at 2026-09-24 21:11:38 on dist-test-slave-c7tk"
server_key: "fe42203acf4e4edab0d12f4dbcb25617"
server_key_iv: "65b4f264956bd85e85de7378624afd27"
server_key_version: "encryptionkey@0"
I20260924 21:11:38.897398 18633 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260924 21:11:38.898149 18648 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 21:11:38.898336 18633 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 21:11:38.898425 18633 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790284076800039-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790284076800039-392-0/minicluster-data/ts-1/wal
uuid: "182c45293918451a9820fb5399eb7443"
format_stamp: "Formatted at 2026-09-24 21:11:38 on dist-test-slave-c7tk"
server_key: "fe42203acf4e4edab0d12f4dbcb25617"
server_key_iv: "65b4f264956bd85e85de7378624afd27"
server_key_version: "encryptionkey@0"
I20260924 21:11:38.898504 18633 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790284076800039-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790284076800039-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790284076800039-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 21:11:38.905428 18633 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 21:11:38.907258 18633 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 21:11:38.907388 18633 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 21:11:38.907573 18633 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 21:11:38.907811 18633 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 21:11:38.907855 18633 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 21:11:38.907882 18633 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 21:11:38.907905 18633 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 21:11:38.912886 18633 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:36031
I20260924 21:11:38.912963 18761 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:36031 every 8 connection(s)
I20260924 21:11:38.913229 18633 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790284076800039-392-0/minicluster-data/ts-1/data/info.pb
I20260924 21:11:38.915158 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAzNNjS/build/release/bin/kudu as pid 18633
I20260924 21:11:38.915234 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790284076800039-392-0/minicluster-data/ts-1/wal/instance
I20260924 21:11:38.915403 392 external_mini_cluster.cc:1428] Setting key d4680a10e56464f09afb056796987c3d
Sep 24 21:11:38 dist-test-slave-c7tk krb5kdc[18392](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284298, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 21:11:38.921564 18477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:11:38.913772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:34885 (local address 127.0.98.62:33695)
0924 21:11:38.913834 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0924 21:11:38.913836 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 21:11:38.914031 (+ 195us) server_negotiation.cc:407] Connection header received
0924 21:11:38.914787 (+ 756us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:11:38.914789 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:11:38.914800 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:11:38.914845 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:11:38.915386 (+ 541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:11:38.916014 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:11:38.916849 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:11:38.917035 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:11:38.917106 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:11:38.918977 (+ 1871us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 21:11:38.918979 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 21:11:38.918979 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 21:11:38.918987 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 21:11:38.920458 (+ 1471us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:11:38.920788 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:11:38.920789 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:11:38.920789 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:11:38.920843 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:11:38.921032 (+ 189us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:11:38.921033 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:11:38.921033 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:11:38.921149 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 21:11:38.921212 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0924 21:11:38.921404 (+ 192us) server_negotiation.cc:299] Negotiation successful
0924 21:11:38.921476 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260924 21:11:38.921576 18764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:11:38.913766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33695 (local address 127.0.98.2:34885)
0924 21:11:38.913917 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0924 21:11:38.913946 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0924 21:11:38.914694 (+ 748us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 21:11:38.914899 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 21:11:38.914902 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 21:11:38.915113 (+ 211us) client_negotiation.cc:189] Negotiated authn=SASL
0924 21:11:38.915309 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:11:38.915313 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:11:38.916076 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:11:38.916078 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:11:38.916758 (+ 680us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:11:38.916760 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:11:38.917113 (+ 353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:11:38.917114 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:11:38.917224 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:11:38.917595 (+ 371us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 21:11:38.917601 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0924 21:11:38.918895 (+ 1294us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 21:11:38.920538 (+ 1643us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 21:11:38.920542 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 21:11:38.920544 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 21:11:38.920719 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 21:11:38.920903 (+ 184us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 21:11:38.920903 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 21:11:38.920903 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 21:11:38.920978 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 21:11:38.921207 (+ 229us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 21:11:38.921210 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 21:11:38.921297 (+ 87us) client_negotiation.cc:769] Sending connection context
0924 21:11:38.921363 (+ 66us) client_negotiation.cc:240] Negotiation successful
0924 21:11:38.921454 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":96,"threads_started":1}
I20260924 21:11:38.921988 18762 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33695
I20260924 21:11:38.922081 18762 heartbeater.cc:464] Registering TS with master...
I20260924 21:11:38.922232 18762 heartbeater.cc:511] Master 127.0.98.62:33695 requested a full tablet report, sending...
I20260924 21:11:38.922578 18437 ts_manager.cc:194] Registered new tserver with Master: 182c45293918451a9820fb5399eb7443 (127.0.98.2:36031)
I20260924 21:11:38.923041 18437 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:34885
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790284076800039-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790284076800039-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260924 21:11:38.938576 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAzNNjS/build/release/bin/kudu
/tmp/dist-test-taskAzNNjS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790284076800039-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790284076800039-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790284076800039-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790284076800039-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33695
--keytab_file=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790284076800039-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39243
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790284076800039-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790284076800039-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790284076800039-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 21:11:39.010839 18767 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 21:11:39.010991 18767 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 21:11:39.011010 18767 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 21:11:39.012006 18767 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 21:11:39.012040 18767 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 21:11:39.012065 18767 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 21:11:39.013324 18767 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39243
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790284076800039-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790284076800039-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790284076800039-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790284076800039-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33695
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18767
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790284076800039-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 19:49:06 UTC on ccycloud.quasar-vhqjoc.root.comops.site
build id 4670864
I20260924 21:11:39.013484 18767 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 21:11:39.013645 18767 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 21:11:39.017823 18767 server_base.cc:1034] running on GCE node
W20260924 21:11:39.017845 18775 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 21:11:39.017776 18773 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 21:11:39.017784 18772 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 21:11:39.018210 18767 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 21:11:39.018477 18767 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 21:11:39.019631 18767 hybrid_clock.cc:648] HybridClock initialized: now 1790284299019629 us; error 35 us; skew 500 ppm
Sep 24 21:11:39 dist-test-slave-c7tk krb5kdc[18392](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284299, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 21:11:39.021778 18767 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 21:11:39.022193 18767 webserver.cc:460] Webserver started at http://127.0.98.3:38013/ using document root <none> and password file <none>
I20260924 21:11:39.022396 18767 fs_manager.cc:359] Metadata directory not provided
I20260924 21:11:39.022454 18767 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 21:11:39.022578 18767 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 21:11:39.023451 18767 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790284076800039-392-0/minicluster-data/ts-2/data/instance:
uuid: "ecce9bc321c04195919c69bc78346ecb"
format_stamp: "Formatted at 2026-09-24 21:11:39 on dist-test-slave-c7tk"
server_key: "08a8fe78181a397a1ffdad95bb287ac8"
server_key_iv: "3dc4711254832b0931432d016bdf54df"
server_key_version: "encryptionkey@0"
I20260924 21:11:39.023765 18767 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790284076800039-392-0/minicluster-data/ts-2/wal/instance:
uuid: "ecce9bc321c04195919c69bc78346ecb"
format_stamp: "Formatted at 2026-09-24 21:11:39 on dist-test-slave-c7tk"
server_key: "08a8fe78181a397a1ffdad95bb287ac8"
server_key_iv: "3dc4711254832b0931432d016bdf54df"
server_key_version: "encryptionkey@0"
I20260924 21:11:39.024855 18767 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260924 21:11:39.025478 18782 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 21:11:39.025630 18767 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 21:11:39.025712 18767 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790284076800039-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790284076800039-392-0/minicluster-data/ts-2/wal
uuid: "ecce9bc321c04195919c69bc78346ecb"
format_stamp: "Formatted at 2026-09-24 21:11:39 on dist-test-slave-c7tk"
server_key: "08a8fe78181a397a1ffdad95bb287ac8"
server_key_iv: "3dc4711254832b0931432d016bdf54df"
server_key_version: "encryptionkey@0"
I20260924 21:11:39.025787 18767 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790284076800039-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790284076800039-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790284076800039-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 21:11:39.039979 18767 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 21:11:39.041528 18767 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 21:11:39.041661 18767 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 21:11:39.041862 18767 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 21:11:39.042115 18767 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 21:11:39.042161 18767 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 21:11:39.042191 18767 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 21:11:39.042213 18767 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 21:11:39.047057 18767 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:44365
I20260924 21:11:39.047094 18895 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:44365 every 8 connection(s)
I20260924 21:11:39.047348 18767 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790284076800039-392-0/minicluster-data/ts-2/data/info.pb
I20260924 21:11:39.052454 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAzNNjS/build/release/bin/kudu as pid 18767
I20260924 21:11:39.052549 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790284076800039-392-0/minicluster-data/ts-2/wal/instance
I20260924 21:11:39.052704 392 external_mini_cluster.cc:1428] Setting key 2282d4523230135035d787bf910250e2
Sep 24 21:11:39 dist-test-slave-c7tk krb5kdc[18392](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284299, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 21:11:39.055410 18477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:11:39.047973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:44047 (local address 127.0.98.62:33695)
0924 21:11:39.048034 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0924 21:11:39.048036 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 21:11:39.048269 (+ 233us) server_negotiation.cc:407] Connection header received
0924 21:11:39.049236 (+ 967us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:11:39.049236 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:11:39.049246 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:11:39.049300 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:11:39.049841 (+ 541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:11:39.050259 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:11:39.051048 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:11:39.051219 (+ 171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:11:39.051263 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:11:39.053306 (+ 2043us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 21:11:39.053308 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 21:11:39.053308 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 21:11:39.053317 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 21:11:39.054507 (+ 1190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:11:39.054807 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:11:39.054807 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:11:39.054807 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:11:39.054849 (+ 42us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:11:39.055018 (+ 169us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:11:39.055018 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:11:39.055019 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:11:39.055099 (+ 80us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 21:11:39.055140 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0924 21:11:39.055308 (+ 168us) server_negotiation.cc:299] Negotiation successful
0924 21:11:39.055358 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260924 21:11:39.055533 18898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:11:39.047953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33695 (local address 127.0.98.3:44047)
0924 21:11:39.048078 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0924 21:11:39.048126 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0924 21:11:39.049142 (+ 1016us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 21:11:39.049349 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 21:11:39.049352 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 21:11:39.049598 (+ 246us) client_negotiation.cc:189] Negotiated authn=SASL
0924 21:11:39.049746 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:11:39.049749 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:11:39.050323 (+ 574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:11:39.050323 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:11:39.050975 (+ 652us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:11:39.050977 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:11:39.051279 (+ 302us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:11:39.051280 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:11:39.051363 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:11:39.051790 (+ 427us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 21:11:39.051796 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0924 21:11:39.053208 (+ 1412us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 21:11:39.054564 (+ 1356us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 21:11:39.054575 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 21:11:39.054576 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 21:11:39.054749 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 21:11:39.054898 (+ 149us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 21:11:39.054898 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 21:11:39.054898 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 21:11:39.054972 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 21:11:39.055144 (+ 172us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 21:11:39.055146 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 21:11:39.055228 (+ 82us) client_negotiation.cc:769] Sending connection context
0924 21:11:39.055294 (+ 66us) client_negotiation.cc:240] Negotiation successful
0924 21:11:39.055383 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":92,"thread_start_us":68,"threads_started":1}
I20260924 21:11:39.055882 18896 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33695
I20260924 21:11:39.055943 18896 heartbeater.cc:464] Registering TS with master...
I20260924 21:11:39.056044 18896 heartbeater.cc:511] Master 127.0.98.62:33695 requested a full tablet report, sending...
I20260924 21:11:39.056298 18437 ts_manager.cc:194] Registered new tserver with Master: ecce9bc321c04195919c69bc78346ecb (127.0.98.3:44365)
I20260924 21:11:39.056608 18437 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:44047
I20260924 21:11:39.064222 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 21:11:39.070996 18477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:11:39.065412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59550 (local address 127.0.98.62:33695)
0924 21:11:39.065490 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0924 21:11:39.065491 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 21:11:39.065578 (+ 87us) server_negotiation.cc:407] Connection header received
0924 21:11:39.065640 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:11:39.065640 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:11:39.065648 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:11:39.065686 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:11:39.066196 (+ 510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:11:39.066576 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:11:39.067189 (+ 613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:11:39.067361 (+ 172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:11:39.067429 (+ 68us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:11:39.068556 (+ 1127us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 21:11:39.068557 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 21:11:39.068558 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 21:11:39.068564 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 21:11:39.069966 (+ 1402us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:11:39.070319 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:11:39.070320 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:11:39.070320 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:11:39.070365 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:11:39.070559 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:11:39.070560 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:11:39.070560 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:11:39.070671 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 21:11:39.070768 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0924 21:11:39.070868 (+ 100us) server_negotiation.cc:299] Negotiation successful
0924 21:11:39.070918 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260924 21:11:39.072192 18437 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59550:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 21:11:39.073601 18437 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 21:11:39.078801 18907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:11:39.075504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36031 (local address 127.0.0.1:45274)
0924 21:11:39.075703 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0924 21:11:39.075711 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0924 21:11:39.075744 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 21:11:39.076074 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 21:11:39.076075 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 21:11:39.076076 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 21:11:39.076105 (+ 29us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:11:39.076106 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:11:39.077041 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:11:39.077042 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:11:39.077849 (+ 807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:11:39.077852 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:11:39.078551 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:11:39.078551 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:11:39.078619 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:11:39.078631 (+ 12us) client_negotiation.cc:769] Sending connection context
0924 21:11:39.078693 (+ 62us) client_negotiation.cc:240] Negotiation successful
0924 21:11:39.078736 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":187,"mutex_wait_us":53,"thread_start_us":71,"threads_started":1}
I20260924 21:11:39.078831 18909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:11:39.075706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45274 (local address 127.0.98.2:36031)
0924 21:11:39.075862 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0924 21:11:39.075865 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 21:11:39.075890 (+ 25us) server_negotiation.cc:407] Connection header received
0924 21:11:39.075922 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:11:39.075924 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:11:39.075937 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:11:39.076000 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 21:11:39.076171 (+ 171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:11:39.076944 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:11:39.077914 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:11:39.078469 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:11:39.078543 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:11:39.078603 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0924 21:11:39.078714 (+ 111us) server_negotiation.cc:299] Negotiation successful
0924 21:11:39.078751 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":72,"threads_started":1}
I20260924 21:11:39.079314 18905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:11:39.075354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35799 (local address 127.0.0.1:60952)
0924 21:11:39.075479 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0924 21:11:39.075506 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0924 21:11:39.075559 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 21:11:39.076013 (+ 454us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 21:11:39.076014 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 21:11:39.076016 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 21:11:39.076064 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:11:39.076065 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:11:39.077162 (+ 1097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:11:39.077163 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:11:39.078166 (+ 1003us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:11:39.078169 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:11:39.079065 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:11:39.079066 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:11:39.079130 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:11:39.079134 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 21:11:39.079196 (+ 62us) client_negotiation.cc:240] Negotiation successful
0924 21:11:39.079229 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"mutex_wait_us":20,"thread_start_us":48,"threads_started":1}
I20260924 21:11:39.079324 18908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:11:39.075580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60952 (local address 127.0.98.1:35799)
0924 21:11:39.075764 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0924 21:11:39.075768 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 21:11:39.075776 (+ 8us) server_negotiation.cc:407] Connection header received
0924 21:11:39.075812 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:11:39.075814 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:11:39.075830 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:11:39.075926 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 21:11:39.076156 (+ 230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:11:39.077083 (+ 927us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:11:39.078277 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:11:39.078979 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:11:39.079057 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:11:39.079127 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0924 21:11:39.079210 (+ 83us) server_negotiation.cc:299] Negotiation successful
0924 21:11:39.079251 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":157,"thread_start_us":85,"threads_started":1}
I20260924 21:11:39.079667 18904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:11:39.075354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44365 (local address 127.0.0.1:38028)
0924 21:11:39.075638 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0924 21:11:39.075646 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0924 21:11:39.075686 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 21:11:39.075852 (+ 166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 21:11:39.075855 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 21:11:39.075856 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 21:11:39.075933 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:11:39.075935 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:11:39.076996 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:11:39.076997 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:11:39.078384 (+ 1387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:11:39.078386 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:11:39.079502 (+ 1116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:11:39.079502 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:11:39.079539 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:11:39.079541 (+ 2us) client_negotiation.cc:769] Sending connection context
0924 21:11:39.079576 (+ 35us) client_negotiation.cc:240] Negotiation successful
0924 21:11:39.079597 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":274,"mutex_wait_us":43,"thread_start_us":48,"threads_started":1}
I20260924 21:11:39.079594 18696 tablet_service.cc:1467] Processing CreateTablet for tablet ebc7e4319e0148e585e7707279a0fac1 (DEFAULT_TABLE table=test-table [id=10c1f14cf52242f0aefd30ac7b80f3b3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 21:11:39.079903 18696 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ebc7e4319e0148e585e7707279a0fac1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 21:11:39.079927 18906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:11:39.075419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38028 (local address 127.0.98.3:44365)
0924 21:11:39.075550 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0924 21:11:39.075553 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 21:11:39.075737 (+ 184us) server_negotiation.cc:407] Connection header received
0924 21:11:39.075771 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:11:39.075773 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:11:39.075786 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:11:39.075846 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 21:11:39.075982 (+ 136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:11:39.076884 (+ 902us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:11:39.078442 (+ 1558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:11:39.079266 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:11:39.079325 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:11:39.079817 (+ 492us) server_negotiation.cc:1035] Waiting for connection context
0924 21:11:39.079838 (+ 21us) server_negotiation.cc:299] Negotiation successful
0924 21:11:39.079866 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105,"thread_start_us":80,"threads_started":1}
I20260924 21:11:39.079967 18562 tablet_service.cc:1467] Processing CreateTablet for tablet ebc7e4319e0148e585e7707279a0fac1 (DEFAULT_TABLE table=test-table [id=10c1f14cf52242f0aefd30ac7b80f3b3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 21:11:39.080240 18562 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ebc7e4319e0148e585e7707279a0fac1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 21:11:39.080401 18830 tablet_service.cc:1467] Processing CreateTablet for tablet ebc7e4319e0148e585e7707279a0fac1 (DEFAULT_TABLE table=test-table [id=10c1f14cf52242f0aefd30ac7b80f3b3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 21:11:39.080629 18830 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ebc7e4319e0148e585e7707279a0fac1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 21:11:39.082746 18911 tablet_bootstrap.cc:492] T ebc7e4319e0148e585e7707279a0fac1 P 182c45293918451a9820fb5399eb7443: Bootstrap starting.
I20260924 21:11:39.083211 18912 tablet_bootstrap.cc:492] T ebc7e4319e0148e585e7707279a0fac1 P ecce9bc321c04195919c69bc78346ecb: Bootstrap starting.
I20260924 21:11:39.083249 18911 tablet_bootstrap.cc:654] T ebc7e4319e0148e585e7707279a0fac1 P 182c45293918451a9820fb5399eb7443: Neither blocks nor log segments found. Creating new log.
I20260924 21:11:39.083709 18910 tablet_bootstrap.cc:492] T ebc7e4319e0148e585e7707279a0fac1 P 6b94a3c5013941e6bddecd22313cbb6c: Bootstrap starting.
I20260924 21:11:39.083775 18912 tablet_bootstrap.cc:654] T ebc7e4319e0148e585e7707279a0fac1 P ecce9bc321c04195919c69bc78346ecb: Neither blocks nor log segments found. Creating new log.
I20260924 21:11:39.083757 18911 log.cc:824] T ebc7e4319e0148e585e7707279a0fac1 P 182c45293918451a9820fb5399eb7443: Log is configured to *not* fsync() on all Append() calls
I20260924 21:11:39.084257 18912 log.cc:824] T ebc7e4319e0148e585e7707279a0fac1 P ecce9bc321c04195919c69bc78346ecb: Log is configured to *not* fsync() on all Append() calls
I20260924 21:11:39.084300 18910 tablet_bootstrap.cc:654] T ebc7e4319e0148e585e7707279a0fac1 P 6b94a3c5013941e6bddecd22313cbb6c: Neither blocks nor log segments found. Creating new log.
I20260924 21:11:39.084359 18911 tablet_bootstrap.cc:492] T ebc7e4319e0148e585e7707279a0fac1 P 182c45293918451a9820fb5399eb7443: No bootstrap required, opened a new log
I20260924 21:11:39.084420 18911 ts_tablet_manager.cc:1397] T ebc7e4319e0148e585e7707279a0fac1 P 182c45293918451a9820fb5399eb7443: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 21:11:39.084748 18910 log.cc:824] T ebc7e4319e0148e585e7707279a0fac1 P 6b94a3c5013941e6bddecd22313cbb6c: Log is configured to *not* fsync() on all Append() calls
I20260924 21:11:39.084933 18912 tablet_bootstrap.cc:492] T ebc7e4319e0148e585e7707279a0fac1 P ecce9bc321c04195919c69bc78346ecb: No bootstrap required, opened a new log
I20260924 21:11:39.084991 18912 ts_tablet_manager.cc:1397] T ebc7e4319e0148e585e7707279a0fac1 P ecce9bc321c04195919c69bc78346ecb: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 21:11:39.085472 18910 tablet_bootstrap.cc:492] T ebc7e4319e0148e585e7707279a0fac1 P 6b94a3c5013941e6bddecd22313cbb6c: No bootstrap required, opened a new log
I20260924 21:11:39.085539 18910 ts_tablet_manager.cc:1397] T ebc7e4319e0148e585e7707279a0fac1 P 6b94a3c5013941e6bddecd22313cbb6c: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 21:11:39.085986 18911 raft_consensus.cc:348] T ebc7e4319e0148e585e7707279a0fac1 P 182c45293918451a9820fb5399eb7443 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecce9bc321c04195919c69bc78346ecb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44365 } } peers { permanent_uuid: "182c45293918451a9820fb5399eb7443" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36031 } } peers { permanent_uuid: "6b94a3c5013941e6bddecd22313cbb6c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35799 } }
I20260924 21:11:39.086125 18911 raft_consensus.cc:374] T ebc7e4319e0148e585e7707279a0fac1 P 182c45293918451a9820fb5399eb7443 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 21:11:39.086165 18911 raft_consensus.cc:729] T ebc7e4319e0148e585e7707279a0fac1 P 182c45293918451a9820fb5399eb7443 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 182c45293918451a9820fb5399eb7443, State: Initialized, Role: FOLLOWER
I20260924 21:11:39.086293 18911 consensus_queue.cc:260] T ebc7e4319e0148e585e7707279a0fac1 P 182c45293918451a9820fb5399eb7443 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecce9bc321c04195919c69bc78346ecb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44365 } } peers { permanent_uuid: "182c45293918451a9820fb5399eb7443" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36031 } } peers { permanent_uuid: "6b94a3c5013941e6bddecd22313cbb6c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35799 } }
I20260924 21:11:39.086556 18911 ts_tablet_manager.cc:1428] T ebc7e4319e0148e585e7707279a0fac1 P 182c45293918451a9820fb5399eb7443: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 21:11:39.086684 18762 heartbeater.cc:503] Master 127.0.98.62:33695 was elected leader, sending a full tablet report...
I20260924 21:11:39.086719 18912 raft_consensus.cc:348] T ebc7e4319e0148e585e7707279a0fac1 P ecce9bc321c04195919c69bc78346ecb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecce9bc321c04195919c69bc78346ecb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44365 } } peers { permanent_uuid: "182c45293918451a9820fb5399eb7443" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36031 } } peers { permanent_uuid: "6b94a3c5013941e6bddecd22313cbb6c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35799 } }
I20260924 21:11:39.086867 18912 raft_consensus.cc:374] T ebc7e4319e0148e585e7707279a0fac1 P ecce9bc321c04195919c69bc78346ecb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 21:11:39.086908 18912 raft_consensus.cc:729] T ebc7e4319e0148e585e7707279a0fac1 P ecce9bc321c04195919c69bc78346ecb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ecce9bc321c04195919c69bc78346ecb, State: Initialized, Role: FOLLOWER
I20260924 21:11:39.087007 18912 consensus_queue.cc:260] T ebc7e4319e0148e585e7707279a0fac1 P ecce9bc321c04195919c69bc78346ecb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecce9bc321c04195919c69bc78346ecb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44365 } } peers { permanent_uuid: "182c45293918451a9820fb5399eb7443" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36031 } } peers { permanent_uuid: "6b94a3c5013941e6bddecd22313cbb6c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35799 } }
I20260924 21:11:39.087250 18896 heartbeater.cc:503] Master 127.0.98.62:33695 was elected leader, sending a full tablet report...
I20260924 21:11:39.087244 18912 ts_tablet_manager.cc:1428] T ebc7e4319e0148e585e7707279a0fac1 P ecce9bc321c04195919c69bc78346ecb: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 21:11:39.087718 18910 raft_consensus.cc:348] T ebc7e4319e0148e585e7707279a0fac1 P 6b94a3c5013941e6bddecd22313cbb6c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecce9bc321c04195919c69bc78346ecb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44365 } } peers { permanent_uuid: "182c45293918451a9820fb5399eb7443" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36031 } } peers { permanent_uuid: "6b94a3c5013941e6bddecd22313cbb6c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35799 } }
I20260924 21:11:39.087857 18910 raft_consensus.cc:374] T ebc7e4319e0148e585e7707279a0fac1 P 6b94a3c5013941e6bddecd22313cbb6c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 21:11:39.087896 18910 raft_consensus.cc:729] T ebc7e4319e0148e585e7707279a0fac1 P 6b94a3c5013941e6bddecd22313cbb6c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b94a3c5013941e6bddecd22313cbb6c, State: Initialized, Role: FOLLOWER
I20260924 21:11:39.087987 18910 consensus_queue.cc:260] T ebc7e4319e0148e585e7707279a0fac1 P 6b94a3c5013941e6bddecd22313cbb6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecce9bc321c04195919c69bc78346ecb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44365 } } peers { permanent_uuid: "182c45293918451a9820fb5399eb7443" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36031 } } peers { permanent_uuid: "6b94a3c5013941e6bddecd22313cbb6c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35799 } }
I20260924 21:11:39.088217 18910 ts_tablet_manager.cc:1428] T ebc7e4319e0148e585e7707279a0fac1 P 6b94a3c5013941e6bddecd22313cbb6c: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 21:11:39.088289 18628 heartbeater.cc:503] Master 127.0.98.62:33695 was elected leader, sending a full tablet report...
W20260924 21:11:39.163779 18763 tablet.cc:2369] T ebc7e4319e0148e585e7707279a0fac1 P 182c45293918451a9820fb5399eb7443: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 21:11:39.237857 18917 raft_consensus.cc:482] T ebc7e4319e0148e585e7707279a0fac1 P ecce9bc321c04195919c69bc78346ecb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 21:11:39.237955 18917 raft_consensus.cc:504] T ebc7e4319e0148e585e7707279a0fac1 P ecce9bc321c04195919c69bc78346ecb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecce9bc321c04195919c69bc78346ecb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44365 } } peers { permanent_uuid: "182c45293918451a9820fb5399eb7443" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36031 } } peers { permanent_uuid: "6b94a3c5013941e6bddecd22313cbb6c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35799 } }
I20260924 21:11:39.238350 18917 leader_election.cc:288] T ebc7e4319e0148e585e7707279a0fac1 P ecce9bc321c04195919c69bc78346ecb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 182c45293918451a9820fb5399eb7443 (127.0.98.2:36031), 6b94a3c5013941e6bddecd22313cbb6c (127.0.98.1:35799)
I20260924 21:11:39.240820 18908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:11:39.238636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:48155 (local address 127.0.98.1:35799)
0924 21:11:39.238677 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0924 21:11:39.238679 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 21:11:39.238694 (+ 15us) server_negotiation.cc:407] Connection header received
0924 21:11:39.238716 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:11:39.238717 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:11:39.238725 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:11:39.238789 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 21:11:39.238940 (+ 151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:11:39.239424 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:11:39.240225 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:11:39.240528 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:11:39.240598 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:11:39.240607 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0924 21:11:39.240732 (+ 125us) server_negotiation.cc:299] Negotiation successful
0924 21:11:39.240767 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260924 21:11:39.240820 18898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:11:39.238514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35799 (local address 127.0.98.3:48155)
0924 21:11:39.238554 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0924 21:11:39.238566 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 21:11:39.238611 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 21:11:39.238791 (+ 180us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 21:11:39.238792 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 21:11:39.238793 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 21:11:39.238867 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:11:39.238869 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:11:39.239513 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:11:39.239520 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:11:39.240143 (+ 623us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:11:39.240144 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:11:39.240596 (+ 452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:11:39.240597 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:11:39.240667 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:11:39.240670 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 21:11:39.240726 (+ 56us) client_negotiation.cc:240] Negotiation successful
0924 21:11:39.240758 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":20}
I20260924 21:11:39.240952 18919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:11:39.238518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36031 (local address 127.0.98.3:39351)
0924 21:11:39.238700 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0924 21:11:39.238707 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0924 21:11:39.238784 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 21:11:39.238926 (+ 142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 21:11:39.238927 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 21:11:39.238929 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 21:11:39.239000 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:11:39.239001 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:11:39.239609 (+ 608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:11:39.239609 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:11:39.240238 (+ 629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:11:39.240239 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:11:39.240706 (+ 467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:11:39.240707 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:11:39.240773 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:11:39.240775 (+ 2us) client_negotiation.cc:769] Sending connection context
0924 21:11:39.240833 (+ 58us) client_negotiation.cc:240] Negotiation successful
0924 21:11:39.240882 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160,"mutex_wait_us":19,"thread_start_us":92,"threads_started":1}
I20260924 21:11:39.240943 18909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:11:39.238590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:39351 (local address 127.0.98.2:36031)
0924 21:11:39.238641 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0924 21:11:39.238643 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 21:11:39.238774 (+ 131us) server_negotiation.cc:407] Connection header received
0924 21:11:39.238840 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:11:39.238841 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:11:39.238850 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:11:39.238923 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 21:11:39.239070 (+ 147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:11:39.239533 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:11:39.240319 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:11:39.240632 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:11:39.240703 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:11:39.240711 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0924 21:11:39.240838 (+ 127us) server_negotiation.cc:299] Negotiation successful
0924 21:11:39.240887 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260924 21:11:39.241156 18582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ebc7e4319e0148e585e7707279a0fac1" candidate_uuid: "ecce9bc321c04195919c69bc78346ecb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b94a3c5013941e6bddecd22313cbb6c" is_pre_election: true
I20260924 21:11:39.241230 18716 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ebc7e4319e0148e585e7707279a0fac1" candidate_uuid: "ecce9bc321c04195919c69bc78346ecb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "182c45293918451a9820fb5399eb7443" is_pre_election: true
I20260924 21:11:39.241313 18582 raft_consensus.cc:2454] T ebc7e4319e0148e585e7707279a0fac1 P 6b94a3c5013941e6bddecd22313cbb6c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecce9bc321c04195919c69bc78346ecb in term 0.
I20260924 21:11:39.241359 18716 raft_consensus.cc:2454] T ebc7e4319e0148e585e7707279a0fac1 P 182c45293918451a9820fb5399eb7443 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecce9bc321c04195919c69bc78346ecb in term 0.
I20260924 21:11:39.241545 18784 leader_election.cc:302] T ebc7e4319e0148e585e7707279a0fac1 P ecce9bc321c04195919c69bc78346ecb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6b94a3c5013941e6bddecd22313cbb6c, ecce9bc321c04195919c69bc78346ecb; no voters:
I20260924 21:11:39.241677 18917 raft_consensus.cc:2781] T ebc7e4319e0148e585e7707279a0fac1 P ecce9bc321c04195919c69bc78346ecb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 21:11:39.241729 18917 raft_consensus.cc:482] T ebc7e4319e0148e585e7707279a0fac1 P ecce9bc321c04195919c69bc78346ecb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 21:11:39.241745 18917 raft_consensus.cc:3037] T ebc7e4319e0148e585e7707279a0fac1 P ecce9bc321c04195919c69bc78346ecb [term 0 FOLLOWER]: Advancing to term 1
I20260924 21:11:39.242377 18917 raft_consensus.cc:504] T ebc7e4319e0148e585e7707279a0fac1 P ecce9bc321c04195919c69bc78346ecb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecce9bc321c04195919c69bc78346ecb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44365 } } peers { permanent_uuid: "182c45293918451a9820fb5399eb7443" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36031 } } peers { permanent_uuid: "6b94a3c5013941e6bddecd22313cbb6c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35799 } }
I20260924 21:11:39.242544 18917 leader_election.cc:288] T ebc7e4319e0148e585e7707279a0fac1 P ecce9bc321c04195919c69bc78346ecb [CANDIDATE]: Term 1 election: Requested vote from peers 182c45293918451a9820fb5399eb7443 (127.0.98.2:36031), 6b94a3c5013941e6bddecd22313cbb6c (127.0.98.1:35799)
I20260924 21:11:39.242614 18716 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ebc7e4319e0148e585e7707279a0fac1" candidate_uuid: "ecce9bc321c04195919c69bc78346ecb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "182c45293918451a9820fb5399eb7443"
I20260924 21:11:39.242635 18582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ebc7e4319e0148e585e7707279a0fac1" candidate_uuid: "ecce9bc321c04195919c69bc78346ecb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b94a3c5013941e6bddecd22313cbb6c"
I20260924 21:11:39.242702 18716 raft_consensus.cc:3037] T ebc7e4319e0148e585e7707279a0fac1 P 182c45293918451a9820fb5399eb7443 [term 0 FOLLOWER]: Advancing to term 1
I20260924 21:11:39.242694 18582 raft_consensus.cc:3037] T ebc7e4319e0148e585e7707279a0fac1 P 6b94a3c5013941e6bddecd22313cbb6c [term 0 FOLLOWER]: Advancing to term 1
I20260924 21:11:39.243319 18716 raft_consensus.cc:2454] T ebc7e4319e0148e585e7707279a0fac1 P 182c45293918451a9820fb5399eb7443 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ecce9bc321c04195919c69bc78346ecb in term 1.
I20260924 21:11:39.243328 18582 raft_consensus.cc:2454] T ebc7e4319e0148e585e7707279a0fac1 P 6b94a3c5013941e6bddecd22313cbb6c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ecce9bc321c04195919c69bc78346ecb in term 1.
I20260924 21:11:39.243500 18785 leader_election.cc:302] T ebc7e4319e0148e585e7707279a0fac1 P ecce9bc321c04195919c69bc78346ecb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 182c45293918451a9820fb5399eb7443, ecce9bc321c04195919c69bc78346ecb; no voters:
I20260924 21:11:39.243599 18917 raft_consensus.cc:2781] T ebc7e4319e0148e585e7707279a0fac1 P ecce9bc321c04195919c69bc78346ecb [term 1 FOLLOWER]: Leader election won for term 1
I20260924 21:11:39.243741 18917 raft_consensus.cc:686] T ebc7e4319e0148e585e7707279a0fac1 P ecce9bc321c04195919c69bc78346ecb [term 1 LEADER]: Becoming Leader. State: Replica: ecce9bc321c04195919c69bc78346ecb, State: Running, Role: LEADER
I20260924 21:11:39.243826 18917 consensus_queue.cc:237] T ebc7e4319e0148e585e7707279a0fac1 P ecce9bc321c04195919c69bc78346ecb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecce9bc321c04195919c69bc78346ecb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44365 } } peers { permanent_uuid: "182c45293918451a9820fb5399eb7443" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36031 } } peers { permanent_uuid: "6b94a3c5013941e6bddecd22313cbb6c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35799 } }
I20260924 21:11:39.244534 18436 catalog_manager.cc:5495] T ebc7e4319e0148e585e7707279a0fac1 P ecce9bc321c04195919c69bc78346ecb reported cstate change: term changed from 0 to 1, leader changed from <none> to ecce9bc321c04195919c69bc78346ecb (127.0.98.3). New cstate: current_term: 1 leader_uuid: "ecce9bc321c04195919c69bc78346ecb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecce9bc321c04195919c69bc78346ecb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44365 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "182c45293918451a9820fb5399eb7443" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36031 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6b94a3c5013941e6bddecd22313cbb6c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35799 } health_report { overall_health: UNKNOWN } } }
I20260924 21:11:39.258234 18906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:11:39.256260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38040 (local address 127.0.98.3:44365)
0924 21:11:39.256303 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0924 21:11:39.256305 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 21:11:39.256312 (+ 7us) server_negotiation.cc:407] Connection header received
0924 21:11:39.256356 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:11:39.256357 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:11:39.256375 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:11:39.256435 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 21:11:39.256575 (+ 140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:11:39.256929 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:11:39.257566 (+ 637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:11:39.257721 (+ 155us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:11:39.257784 (+ 63us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:11:39.257926 (+ 142us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 21:11:39.258004 (+ 78us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 21:11:39.258105 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0924 21:11:39.258130 (+ 25us) server_negotiation.cc:299] Negotiation successful
0924 21:11:39.258136 (+ 6us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260924 21:11:39.259707 18582 raft_consensus.cc:1261] T ebc7e4319e0148e585e7707279a0fac1 P 6b94a3c5013941e6bddecd22313cbb6c [term 1 FOLLOWER]: Refusing update from remote peer ecce9bc321c04195919c69bc78346ecb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 21:11:39.259716 18716 raft_consensus.cc:1261] T ebc7e4319e0148e585e7707279a0fac1 P 182c45293918451a9820fb5399eb7443 [term 1 FOLLOWER]: Refusing update from remote peer ecce9bc321c04195919c69bc78346ecb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 21:11:39.260017 18920 consensus_queue.cc:1035] T ebc7e4319e0148e585e7707279a0fac1 P ecce9bc321c04195919c69bc78346ecb [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b94a3c5013941e6bddecd22313cbb6c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35799 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 21:11:39.260145 18917 consensus_queue.cc:1035] T ebc7e4319e0148e585e7707279a0fac1 P ecce9bc321c04195919c69bc78346ecb [LEADER]: Connected to new peer: Peer: permanent_uuid: "182c45293918451a9820fb5399eb7443" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36031 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 21:11:39.262311 18925 mvcc.cc:205] Tried to move back new op lower bound from 7333004489765576704 to 7333004489703337984. Current Snapshot: MvccSnapshot[applied={T|T < 7333004489765576704}]
I20260924 21:11:39.263712 18924 mvcc.cc:205] Tried to move back new op lower bound from 7333004489765576704 to 7333004489703337984. Current Snapshot: MvccSnapshot[applied={T|T < 7333004489765576704 or (T in {7333004489765576704})}]
I20260924 21:11:39.264099 18437 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59550:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 21:11:39.264173 18437 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59550:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 21:11:39.264783 18437 catalog_manager.cc:5782] T 00000000000000000000000000000000 P dd2bd66449f741dfb7d4214ae641cad9: Sending DeleteTablet for 3 replicas of tablet ebc7e4319e0148e585e7707279a0fac1
I20260924 21:11:39.265064 18562 tablet_service.cc:1514] Processing DeleteTablet for tablet ebc7e4319e0148e585e7707279a0fac1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 21:11:39 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:60952
I20260924 21:11:39.265091 18696 tablet_service.cc:1514] Processing DeleteTablet for tablet ebc7e4319e0148e585e7707279a0fac1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 21:11:39 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:45274
I20260924 21:11:39.265281 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAzNNjS/build/release/bin/kudu with pid 18499
I20260924 21:11:39.265385 18830 tablet_service.cc:1514] Processing DeleteTablet for tablet ebc7e4319e0148e585e7707279a0fac1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 21:11:39 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:38028
I20260924 21:11:39.265780 18933 tablet_replica.cc:330] T ebc7e4319e0148e585e7707279a0fac1 P 182c45293918451a9820fb5399eb7443: stopping tablet replica
I20260924 21:11:39.265882 18933 raft_consensus.cc:2229] T ebc7e4319e0148e585e7707279a0fac1 P 182c45293918451a9820fb5399eb7443 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 21:11:39.265964 18933 raft_consensus.cc:2258] T ebc7e4319e0148e585e7707279a0fac1 P 182c45293918451a9820fb5399eb7443 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 21:11:39.266317 18933 ts_tablet_manager.cc:1905] T ebc7e4319e0148e585e7707279a0fac1 P 182c45293918451a9820fb5399eb7443: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 21:11:39.267634 18933 ts_tablet_manager.cc:1918] T ebc7e4319e0148e585e7707279a0fac1 P 182c45293918451a9820fb5399eb7443: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 21:11:39.267695 18933 log.cc:1196] T ebc7e4319e0148e585e7707279a0fac1 P 182c45293918451a9820fb5399eb7443: Deleting WAL directory at /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790284076800039-392-0/minicluster-data/ts-1/wal/wals/ebc7e4319e0148e585e7707279a0fac1
I20260924 21:11:39.267951 18933 ts_tablet_manager.cc:1939] T ebc7e4319e0148e585e7707279a0fac1 P 182c45293918451a9820fb5399eb7443: Deleting consensus metadata
I20260924 21:11:39.268332 18423 catalog_manager.cc:4841] TS 182c45293918451a9820fb5399eb7443 (127.0.98.2:36031): tablet ebc7e4319e0148e585e7707279a0fac1 (table test-table [id=10c1f14cf52242f0aefd30ac7b80f3b3]) successfully deleted
I20260924 21:11:39.269896 18935 tablet_replica.cc:330] T ebc7e4319e0148e585e7707279a0fac1 P ecce9bc321c04195919c69bc78346ecb: stopping tablet replica
I20260924 21:11:39.269984 18935 raft_consensus.cc:2229] T ebc7e4319e0148e585e7707279a0fac1 P ecce9bc321c04195919c69bc78346ecb [term 1 LEADER]: Raft consensus shutting down.
I20260924 21:11:39.270059 18935 raft_consensus.cc:2258] T ebc7e4319e0148e585e7707279a0fac1 P ecce9bc321c04195919c69bc78346ecb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 21:11:39.270370 18935 ts_tablet_manager.cc:1905] T ebc7e4319e0148e585e7707279a0fac1 P ecce9bc321c04195919c69bc78346ecb: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260924 21:11:39.271777 18422 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:35799 (error 108)
I20260924 21:11:39.271992 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAzNNjS/build/release/bin/kudu with pid 18633
I20260924 21:11:39.272053 18935 ts_tablet_manager.cc:1918] T ebc7e4319e0148e585e7707279a0fac1 P ecce9bc321c04195919c69bc78346ecb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 21:11:39.272109 18904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:11:39.272019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35799 (local address 127.0.0.1:60960)
0924 21:11:39.272056 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0924 21:11:39.272084 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:35799: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
I20260924 21:11:39.272120 18935 log.cc:1196] T ebc7e4319e0148e585e7707279a0fac1 P ecce9bc321c04195919c69bc78346ecb: Deleting WAL directory at /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790284076800039-392-0/minicluster-data/ts-2/wal/wals/ebc7e4319e0148e585e7707279a0fac1
W20260924 21:11:39.272720 18422 catalog_manager.cc:4584] TS 6b94a3c5013941e6bddecd22313cbb6c (127.0.98.1:35799): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ebc7e4319e0148e585e7707279a0fac1: Network error: Client connection negotiation failed: client connection to 127.0.98.1:35799: connect: Connection refused (error 111)
I20260924 21:11:39.276234 18935 ts_tablet_manager.cc:1939] T ebc7e4319e0148e585e7707279a0fac1 P ecce9bc321c04195919c69bc78346ecb: Deleting consensus metadata
I20260924 21:11:39.276252 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAzNNjS/build/release/bin/kudu with pid 18767
I20260924 21:11:39.280750 18904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:11:39.280632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44365 (local address 127.0.0.1:38042)
0924 21:11:39.280690 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0924 21:11:39.280717 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.3:44365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260924 21:11:39.280879 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAzNNjS/build/release/bin/kudu with pid 18405
2026-09-24T21:11:39Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (1842 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-24T21:11:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T21:11:39Z Disabled control of system clock
I20260924 21:11:39.305753 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAzNNjS/build/release/bin/kudu
/tmp/dist-test-taskAzNNjS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790284076800039-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790284076800039-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790284076800039-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790284076800039-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33511
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:39959
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33511
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 21:11:39.374990 18940 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 21:11:39.375159 18940 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 21:11:39.375175 18940 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 21:11:39.376060 18940 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 21:11:39.376089 18940 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 21:11:39.376101 18940 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 21:11:39.376111 18940 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 21:11:39.376121 18940 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 21:11:39.377202 18940 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39959
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790284076800039-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790284076800039-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33511
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33511
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790284076800039-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18940
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790284076800039-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 19:49:06 UTC on ccycloud.quasar-vhqjoc.root.comops.site
build id 4670864
I20260924 21:11:39.377352 18940 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 21:11:39.377534 18940 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 21:11:39.381717 18946 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 21:11:39.381731 18945 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 21:11:39.381733 18948 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 21:11:39.381837 18940 server_base.cc:1034] running on GCE node
I20260924 21:11:39.382174 18940 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 21:11:39.382400 18940 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 21:11:39.383533 18940 hybrid_clock.cc:648] HybridClock initialized: now 1790284299383516 us; error 30 us; skew 500 ppm
I20260924 21:11:39.384622 18940 webserver.cc:460] Webserver started at http://127.0.98.62:33867/ using document root <none> and password file <none>
I20260924 21:11:39.384822 18940 fs_manager.cc:359] Metadata directory not provided
I20260924 21:11:39.384862 18940 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 21:11:39.385003 18940 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 21:11:39.385895 18940 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790284076800039-392-0/minicluster-data/master-0/data/instance:
uuid: "b1dc4d7e0125486296c0bf6497347786"
format_stamp: "Formatted at 2026-09-24 21:11:39 on dist-test-slave-c7tk"
server_key: "07490706704485fec8065ee5eae1b2a4"
server_key_iv: "39753ff3451f47cd072e44c17086cb0d"
server_key_version: "encryptionkey@0"
I20260924 21:11:39.386189 18940 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790284076800039-392-0/minicluster-data/master-0/wal/instance:
uuid: "b1dc4d7e0125486296c0bf6497347786"
format_stamp: "Formatted at 2026-09-24 21:11:39 on dist-test-slave-c7tk"
server_key: "07490706704485fec8065ee5eae1b2a4"
server_key_iv: "39753ff3451f47cd072e44c17086cb0d"
server_key_version: "encryptionkey@0"
I20260924 21:11:39.387270 18940 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 21:11:39.387907 18954 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 21:11:39.388038 18940 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 21:11:39.388146 18940 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790284076800039-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790284076800039-392-0/minicluster-data/master-0/wal
uuid: "b1dc4d7e0125486296c0bf6497347786"
format_stamp: "Formatted at 2026-09-24 21:11:39 on dist-test-slave-c7tk"
server_key: "07490706704485fec8065ee5eae1b2a4"
server_key_iv: "39753ff3451f47cd072e44c17086cb0d"
server_key_version: "encryptionkey@0"
I20260924 21:11:39.388244 18940 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790284076800039-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790284076800039-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790284076800039-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 21:11:39.401224 18940 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 21:11:39.401461 18940 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 21:11:39.401592 18940 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 21:11:39.405012 18940 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33511
I20260924 21:11:39.405068 19006 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33511 every 8 connection(s)
I20260924 21:11:39.405328 18940 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790284076800039-392-0/minicluster-data/master-0/data/info.pb
I20260924 21:11:39.406149 19007 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 21:11:39.408082 19007 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b1dc4d7e0125486296c0bf6497347786: Bootstrap starting.
I20260924 21:11:39.408638 19007 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b1dc4d7e0125486296c0bf6497347786: Neither blocks nor log segments found. Creating new log.
I20260924 21:11:39.408926 19007 log.cc:824] T 00000000000000000000000000000000 P b1dc4d7e0125486296c0bf6497347786: Log is configured to *not* fsync() on all Append() calls
I20260924 21:11:39.408978 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAzNNjS/build/release/bin/kudu as pid 18940
I20260924 21:11:39.409086 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790284076800039-392-0/minicluster-data/master-0/wal/instance
I20260924 21:11:39.409284 392 external_mini_cluster.cc:1428] Setting key 2d632d2c5a6eafd4e22c74cfc0cb988e
I20260924 21:11:39.409662 19007 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b1dc4d7e0125486296c0bf6497347786: No bootstrap required, opened a new log
I20260924 21:11:39.412134 19007 raft_consensus.cc:348] T 00000000000000000000000000000000 P b1dc4d7e0125486296c0bf6497347786 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1dc4d7e0125486296c0bf6497347786" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33511 } }
I20260924 21:11:39.412271 19007 raft_consensus.cc:374] T 00000000000000000000000000000000 P b1dc4d7e0125486296c0bf6497347786 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 21:11:39.412297 19007 raft_consensus.cc:729] T 00000000000000000000000000000000 P b1dc4d7e0125486296c0bf6497347786 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b1dc4d7e0125486296c0bf6497347786, State: Initialized, Role: FOLLOWER
I20260924 21:11:39.412386 19007 consensus_queue.cc:260] T 00000000000000000000000000000000 P b1dc4d7e0125486296c0bf6497347786 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1dc4d7e0125486296c0bf6497347786" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33511 } }
I20260924 21:11:39.412441 19007 raft_consensus.cc:388] T 00000000000000000000000000000000 P b1dc4d7e0125486296c0bf6497347786 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 21:11:39.412467 19007 raft_consensus.cc:482] T 00000000000000000000000000000000 P b1dc4d7e0125486296c0bf6497347786 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 21:11:39.412508 19007 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b1dc4d7e0125486296c0bf6497347786 [term 0 FOLLOWER]: Advancing to term 1
I20260924 21:11:39.413025 19007 raft_consensus.cc:504] T 00000000000000000000000000000000 P b1dc4d7e0125486296c0bf6497347786 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1dc4d7e0125486296c0bf6497347786" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33511 } }
I20260924 21:11:39.413102 19007 leader_election.cc:302] T 00000000000000000000000000000000 P b1dc4d7e0125486296c0bf6497347786 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b1dc4d7e0125486296c0bf6497347786; no voters:
I20260924 21:11:39.413312 19007 leader_election.cc:288] T 00000000000000000000000000000000 P b1dc4d7e0125486296c0bf6497347786 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 21:11:39.413364 19012 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b1dc4d7e0125486296c0bf6497347786 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 21:11:39.413503 19012 raft_consensus.cc:686] T 00000000000000000000000000000000 P b1dc4d7e0125486296c0bf6497347786 [term 1 LEADER]: Becoming Leader. State: Replica: b1dc4d7e0125486296c0bf6497347786, State: Running, Role: LEADER
I20260924 21:11:39.413542 19007 sys_catalog.cc:564] T 00000000000000000000000000000000 P b1dc4d7e0125486296c0bf6497347786 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 21:11:39.413591 19012 consensus_queue.cc:237] T 00000000000000000000000000000000 P b1dc4d7e0125486296c0bf6497347786 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1dc4d7e0125486296c0bf6497347786" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33511 } }
I20260924 21:11:39.413897 19013 sys_catalog.cc:455] T 00000000000000000000000000000000 P b1dc4d7e0125486296c0bf6497347786 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b1dc4d7e0125486296c0bf6497347786" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1dc4d7e0125486296c0bf6497347786" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33511 } } }
I20260924 21:11:39.413991 19013 sys_catalog.cc:458] T 00000000000000000000000000000000 P b1dc4d7e0125486296c0bf6497347786 [sys.catalog]: This master's current role is: LEADER
I20260924 21:11:39.413897 19014 sys_catalog.cc:455] T 00000000000000000000000000000000 P b1dc4d7e0125486296c0bf6497347786 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b1dc4d7e0125486296c0bf6497347786. Latest consensus state: current_term: 1 leader_uuid: "b1dc4d7e0125486296c0bf6497347786" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1dc4d7e0125486296c0bf6497347786" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33511 } } }
I20260924 21:11:39.414212 19014 sys_catalog.cc:458] T 00000000000000000000000000000000 P b1dc4d7e0125486296c0bf6497347786 [sys.catalog]: This master's current role is: LEADER
I20260924 21:11:39.414309 19019 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 21:11:39.415037 19019 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 21:11:39.415643 19011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:11:39.409765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54988 (local address 127.0.98.62:33511)
0924 21:11:39.409995 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0924 21:11:39.410002 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 21:11:39.410039 (+ 37us) server_negotiation.cc:407] Connection header received
0924 21:11:39.410854 (+ 815us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:11:39.410862 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:11:39.410928 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:11:39.411072 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:11:39.411215 (+ 143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:11:39.412454 (+ 1239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:11:39.413304 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:11:39.413915 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:11:39.414367 (+ 452us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:11:39.414751 (+ 384us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 21:11:39.414755 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 21:11:39.414755 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 21:11:39.414970 (+ 215us) server_negotiation.cc:922] calling sasl_server_start()
0924 21:11:39.415015 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 21:11:39.415038 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 21:11:39.415094 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0924 21:11:39.415359 (+ 265us) server_negotiation.cc:299] Negotiation successful
0924 21:11:39.415433 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":164,"thread_start_us":105,"threads_started":1}
I20260924 21:11:39.416309 19019 catalog_manager.cc:1347] Generated new cluster ID: 0194c6d5e2b0442f8ed0c5d8c39e331e
I20260924 21:11:39.416349 19019 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 21:11:39.430608 19019 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 21:11:39.431041 19019 catalog_manager.cc:1504] Loading token signing keys...
I20260924 21:11:39.437958 19019 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b1dc4d7e0125486296c0bf6497347786: Generated new TSK 0
I20260924 21:11:39.438104 19019 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 21:11:39.443125 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAzNNjS/build/release/bin/kudu
/tmp/dist-test-taskAzNNjS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790284076800039-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790284076800039-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790284076800039-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790284076800039-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33511
--builtin_ntp_servers=127.0.98.20:39959
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 21:11:39.523522 19031 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 21:11:39.523682 19031 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 21:11:39.523700 19031 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 21:11:39.524713 19031 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 21:11:39.524746 19031 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 21:11:39.524767 19031 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 21:11:39.526275 19031 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39959
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790284076800039-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790284076800039-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790284076800039-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33511
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19031
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790284076800039-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 19:49:06 UTC on ccycloud.quasar-vhqjoc.root.comops.site
build id 4670864
I20260924 21:11:39.526508 19031 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 21:11:39.526790 19031 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 21:11:39.531558 19039 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 21:11:39.531597 19036 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 21:11:39.531749 19031 server_base.cc:1034] running on GCE node
W20260924 21:11:39.531572 19037 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 21:11:39.532019 19031 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 21:11:39.532250 19031 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 21:11:39.533406 19031 hybrid_clock.cc:648] HybridClock initialized: now 1790284299533392 us; error 30 us; skew 500 ppm
I20260924 21:11:39.534745 19031 webserver.cc:460] Webserver started at http://127.0.98.1:40409/ using document root <none> and password file <none>
I20260924 21:11:39.534960 19031 fs_manager.cc:359] Metadata directory not provided
I20260924 21:11:39.535007 19031 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 21:11:39.535113 19031 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 21:11:39.536152 19031 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790284076800039-392-0/minicluster-data/ts-0/data/instance:
uuid: "ea57f4697be641f19f5d643b4e13c6b2"
format_stamp: "Formatted at 2026-09-24 21:11:39 on dist-test-slave-c7tk"
server_key: "529acffc7b13c6a0b41c6adabfbda016"
server_key_iv: "6a76d046ed7eac1f7bc56441295340b2"
server_key_version: "encryptionkey@0"
I20260924 21:11:39.536491 19031 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790284076800039-392-0/minicluster-data/ts-0/wal/instance:
uuid: "ea57f4697be641f19f5d643b4e13c6b2"
format_stamp: "Formatted at 2026-09-24 21:11:39 on dist-test-slave-c7tk"
server_key: "529acffc7b13c6a0b41c6adabfbda016"
server_key_iv: "6a76d046ed7eac1f7bc56441295340b2"
server_key_version: "encryptionkey@0"
I20260924 21:11:39.537658 19031 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 21:11:39.538362 19045 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 21:11:39.538542 19031 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 21:11:39.538630 19031 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790284076800039-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790284076800039-392-0/minicluster-data/ts-0/wal
uuid: "ea57f4697be641f19f5d643b4e13c6b2"
format_stamp: "Formatted at 2026-09-24 21:11:39 on dist-test-slave-c7tk"
server_key: "529acffc7b13c6a0b41c6adabfbda016"
server_key_iv: "6a76d046ed7eac1f7bc56441295340b2"
server_key_version: "encryptionkey@0"
I20260924 21:11:39.538686 19031 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790284076800039-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790284076800039-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790284076800039-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 21:11:39.549309 19031 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 21:11:39.549541 19031 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 21:11:39.549619 19031 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 21:11:39.549774 19031 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 21:11:39.550065 19031 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 21:11:39.550096 19031 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 21:11:39.550117 19031 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 21:11:39.550135 19031 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 21:11:39.555332 19031 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:36899
I20260924 21:11:39.555373 19158 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:36899 every 8 connection(s)
I20260924 21:11:39.555642 19031 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790284076800039-392-0/minicluster-data/ts-0/data/info.pb
I20260924 21:11:39.557587 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAzNNjS/build/release/bin/kudu as pid 19031
I20260924 21:11:39.557657 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790284076800039-392-0/minicluster-data/ts-0/wal/instance
I20260924 21:11:39.557889 392 external_mini_cluster.cc:1428] Setting key 78b0e5d65139ec8a9e3640f095978a3c
I20260924 21:11:39.558698 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAzNNjS/build/release/bin/kudu
/tmp/dist-test-taskAzNNjS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790284076800039-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790284076800039-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790284076800039-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790284076800039-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33511
--builtin_ntp_servers=127.0.98.20:39959
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260924 21:11:39.560374 19011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:11:39.556237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:58223 (local address 127.0.98.62:33511)
0924 21:11:39.556274 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0924 21:11:39.556276 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 21:11:39.556442 (+ 166us) server_negotiation.cc:407] Connection header received
0924 21:11:39.556804 (+ 362us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:11:39.556805 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:11:39.556815 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:11:39.556869 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:11:39.557387 (+ 518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:11:39.557968 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:11:39.559066 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:11:39.559279 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:11:39.559369 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:11:39.560135 (+ 766us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 21:11:39.560136 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 21:11:39.560137 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 21:11:39.560141 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 21:11:39.560156 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 21:11:39.560160 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 21:11:39.560211 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0924 21:11:39.560283 (+ 72us) server_negotiation.cc:299] Negotiation successful
0924 21:11:39.560305 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260924 21:11:39.560592 19161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:11:39.556198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33511 (local address 127.0.98.1:58223)
0924 21:11:39.556349 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0924 21:11:39.556380 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0924 21:11:39.556733 (+ 353us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 21:11:39.556927 (+ 194us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 21:11:39.556929 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 21:11:39.556936 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0924 21:11:39.557318 (+ 382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:11:39.557321 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:11:39.558094 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:11:39.558095 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:11:39.558959 (+ 864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:11:39.558962 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:11:39.559379 (+ 417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:11:39.559380 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:11:39.559527 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:11:39.559994 (+ 467us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 21:11:39.559994 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0924 21:11:39.560018 (+ 24us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 21:11:39.560037 (+ 19us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 21:11:39.560213 (+ 176us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 21:11:39.560217 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 21:11:39.560220 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 21:11:39.560284 (+ 64us) client_negotiation.cc:240] Negotiation successful
0924 21:11:39.560369 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":80,"threads_started":1}
I20260924 21:11:39.560998 19159 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33511
I20260924 21:11:39.561087 19159 heartbeater.cc:464] Registering TS with master...
I20260924 21:11:39.561236 19159 heartbeater.cc:511] Master 127.0.98.62:33511 requested a full tablet report, sending...
I20260924 21:11:39.561654 18971 ts_manager.cc:194] Registered new tserver with Master: ea57f4697be641f19f5d643b4e13c6b2 (127.0.98.1:36899)
I20260924 21:11:39.562213 18971 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:58223
W20260924 21:11:39.629623 19162 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 21:11:39.629815 19162 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 21:11:39.629874 19162 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 21:11:39.630954 19162 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 21:11:39.631001 19162 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 21:11:39.631022 19162 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 21:11:39.632138 19162 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39959
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790284076800039-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790284076800039-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790284076800039-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33511
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19162
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790284076800039-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 19:49:06 UTC on ccycloud.quasar-vhqjoc.root.comops.site
build id 4670864
I20260924 21:11:39.632329 19162 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 21:11:39.632581 19162 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 21:11:39.636893 19170 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 21:11:39.636961 19168 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 21:11:39.637027 19162 server_base.cc:1034] running on GCE node
W20260924 21:11:39.637890 19167 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 21:11:39.638134 19162 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 21:11:39.638342 19162 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 21:11:39.639479 19162 hybrid_clock.cc:648] HybridClock initialized: now 1790284299639469 us; error 29 us; skew 500 ppm
I20260924 21:11:39.640549 19162 webserver.cc:460] Webserver started at http://127.0.98.2:40545/ using document root <none> and password file <none>
I20260924 21:11:39.640770 19162 fs_manager.cc:359] Metadata directory not provided
I20260924 21:11:39.640834 19162 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 21:11:39.640972 19162 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 21:11:39.641952 19162 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790284076800039-392-0/minicluster-data/ts-1/data/instance:
uuid: "a0c6df6d27fd49c9a14ebfeb586d7bc1"
format_stamp: "Formatted at 2026-09-24 21:11:39 on dist-test-slave-c7tk"
server_key: "7c42c64e44d5e1c58ad6979eda83f837"
server_key_iv: "286ba8bc0190ece4b3ce94ee471511ff"
server_key_version: "encryptionkey@0"
I20260924 21:11:39.642275 19162 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790284076800039-392-0/minicluster-data/ts-1/wal/instance:
uuid: "a0c6df6d27fd49c9a14ebfeb586d7bc1"
format_stamp: "Formatted at 2026-09-24 21:11:39 on dist-test-slave-c7tk"
server_key: "7c42c64e44d5e1c58ad6979eda83f837"
server_key_iv: "286ba8bc0190ece4b3ce94ee471511ff"
server_key_version: "encryptionkey@0"
I20260924 21:11:39.643467 19162 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 21:11:39.644048 19176 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 21:11:39.644193 19162 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 21:11:39.644279 19162 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790284076800039-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790284076800039-392-0/minicluster-data/ts-1/wal
uuid: "a0c6df6d27fd49c9a14ebfeb586d7bc1"
format_stamp: "Formatted at 2026-09-24 21:11:39 on dist-test-slave-c7tk"
server_key: "7c42c64e44d5e1c58ad6979eda83f837"
server_key_iv: "286ba8bc0190ece4b3ce94ee471511ff"
server_key_version: "encryptionkey@0"
I20260924 21:11:39.644351 19162 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790284076800039-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790284076800039-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790284076800039-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 21:11:39.668479 19162 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 21:11:39.668747 19162 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 21:11:39.668895 19162 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 21:11:39.669116 19162 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 21:11:39.669423 19162 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 21:11:39.669495 19162 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 21:11:39.669524 19162 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 21:11:39.669548 19162 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 21:11:39.675098 19162 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:43025
I20260924 21:11:39.675143 19289 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:43025 every 8 connection(s)
I20260924 21:11:39.675410 19162 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790284076800039-392-0/minicluster-data/ts-1/data/info.pb
I20260924 21:11:39.679680 19011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:11:39.676025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:41693 (local address 127.0.98.62:33511)
0924 21:11:39.676081 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0924 21:11:39.676084 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 21:11:39.676291 (+ 207us) server_negotiation.cc:407] Connection header received
0924 21:11:39.676673 (+ 382us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:11:39.676674 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:11:39.676683 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:11:39.676728 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:11:39.677272 (+ 544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:11:39.677685 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:11:39.678589 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:11:39.678773 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:11:39.678828 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:11:39.679517 (+ 689us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 21:11:39.679518 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 21:11:39.679519 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 21:11:39.679522 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0924 21:11:39.679535 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 21:11:39.679538 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 21:11:39.679574 (+ 36us) server_negotiation.cc:1035] Waiting for connection context
0924 21:11:39.679614 (+ 40us) server_negotiation.cc:299] Negotiation successful
0924 21:11:39.679631 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260924 21:11:39.679836 19292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:11:39.676032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33511 (local address 127.0.98.2:41693)
0924 21:11:39.676190 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0924 21:11:39.676217 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0924 21:11:39.676598 (+ 381us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 21:11:39.676770 (+ 172us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 21:11:39.676772 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 21:11:39.676780 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0924 21:11:39.677212 (+ 432us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:11:39.677215 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:11:39.677752 (+ 537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:11:39.677753 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:11:39.678516 (+ 763us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:11:39.678518 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:11:39.678847 (+ 329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:11:39.678847 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:11:39.678948 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:11:39.679390 (+ 442us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 21:11:39.679390 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0924 21:11:39.679414 (+ 24us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 21:11:39.679433 (+ 19us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 21:11:39.679574 (+ 141us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 21:11:39.679576 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 21:11:39.679578 (+ 2us) client_negotiation.cc:769] Sending connection context
0924 21:11:39.679615 (+ 37us) client_negotiation.cc:240] Negotiation successful
0924 21:11:39.679673 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":89,"threads_started":1}
I20260924 21:11:39.680330 19290 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33511
I20260924 21:11:39.680408 19290 heartbeater.cc:464] Registering TS with master...
I20260924 21:11:39.680523 19290 heartbeater.cc:511] Master 127.0.98.62:33511 requested a full tablet report, sending...
I20260924 21:11:39.680837 18971 ts_manager.cc:194] Registered new tserver with Master: a0c6df6d27fd49c9a14ebfeb586d7bc1 (127.0.98.2:43025)
I20260924 21:11:39.681145 18971 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:41693
I20260924 21:11:39.683058 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAzNNjS/build/release/bin/kudu as pid 19162
I20260924 21:11:39.683130 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790284076800039-392-0/minicluster-data/ts-1/wal/instance
I20260924 21:11:39.683262 392 external_mini_cluster.cc:1428] Setting key 5668ec646effcbefa0fcbdb4f0a9d21d
I20260924 21:11:39.683941 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAzNNjS/build/release/bin/kudu
/tmp/dist-test-taskAzNNjS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790284076800039-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790284076800039-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790284076800039-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790284076800039-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33511
--builtin_ntp_servers=127.0.98.20:39959
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 21:11:39.755142 19293 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 21:11:39.755298 19293 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 21:11:39.755316 19293 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 21:11:39.756204 19293 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 21:11:39.756251 19293 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 21:11:39.756273 19293 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 21:11:39.757542 19293 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39959
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790284076800039-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790284076800039-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790284076800039-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33511
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19293
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790284076800039-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 19:49:06 UTC on ccycloud.quasar-vhqjoc.root.comops.site
build id 4670864
I20260924 21:11:39.757712 19293 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 21:11:39.757983 19293 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 21:11:39.762244 19299 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 21:11:39.762306 19293 server_base.cc:1034] running on GCE node
W20260924 21:11:39.762249 19301 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 21:11:39.762262 19298 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 21:11:39.762595 19293 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 21:11:39.762810 19293 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 21:11:39.763957 19293 hybrid_clock.cc:648] HybridClock initialized: now 1790284299763952 us; error 35 us; skew 500 ppm
I20260924 21:11:39.764948 19293 webserver.cc:460] Webserver started at http://127.0.98.3:40701/ using document root <none> and password file <none>
I20260924 21:11:39.765161 19293 fs_manager.cc:359] Metadata directory not provided
I20260924 21:11:39.765219 19293 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 21:11:39.765328 19293 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 21:11:39.766250 19293 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790284076800039-392-0/minicluster-data/ts-2/data/instance:
uuid: "9f90a26f3d364113a853649458ac521d"
format_stamp: "Formatted at 2026-09-24 21:11:39 on dist-test-slave-c7tk"
server_key: "f7aaa2ff207da9bd3cf143052ad598ef"
server_key_iv: "a12895e4e8dcf3644c108df6037d8583"
server_key_version: "encryptionkey@0"
I20260924 21:11:39.766558 19293 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790284076800039-392-0/minicluster-data/ts-2/wal/instance:
uuid: "9f90a26f3d364113a853649458ac521d"
format_stamp: "Formatted at 2026-09-24 21:11:39 on dist-test-slave-c7tk"
server_key: "f7aaa2ff207da9bd3cf143052ad598ef"
server_key_iv: "a12895e4e8dcf3644c108df6037d8583"
server_key_version: "encryptionkey@0"
I20260924 21:11:39.767604 19293 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 21:11:39.768180 19307 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 21:11:39.768370 19293 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 21:11:39.768460 19293 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790284076800039-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790284076800039-392-0/minicluster-data/ts-2/wal
uuid: "9f90a26f3d364113a853649458ac521d"
format_stamp: "Formatted at 2026-09-24 21:11:39 on dist-test-slave-c7tk"
server_key: "f7aaa2ff207da9bd3cf143052ad598ef"
server_key_iv: "a12895e4e8dcf3644c108df6037d8583"
server_key_version: "encryptionkey@0"
I20260924 21:11:39.768544 19293 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790284076800039-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790284076800039-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790284076800039-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 21:11:39.784294 19293 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 21:11:39.784508 19293 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 21:11:39.784624 19293 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 21:11:39.784799 19293 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 21:11:39.785063 19293 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 21:11:39.785123 19293 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 21:11:39.785156 19293 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 21:11:39.785178 19293 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 21:11:39.790337 19293 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:44255
I20260924 21:11:39.790369 19420 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:44255 every 8 connection(s)
I20260924 21:11:39.790709 19293 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790284076800039-392-0/minicluster-data/ts-2/data/info.pb
I20260924 21:11:39.795270 19011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:11:39.791342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:46831 (local address 127.0.98.62:33511)
0924 21:11:39.791403 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0924 21:11:39.791405 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 21:11:39.791562 (+ 157us) server_negotiation.cc:407] Connection header received
0924 21:11:39.791981 (+ 419us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:11:39.791982 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:11:39.791994 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:11:39.792070 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:11:39.792630 (+ 560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:11:39.793105 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:11:39.794103 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:11:39.794283 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:11:39.794358 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:11:39.795030 (+ 672us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 21:11:39.795031 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 21:11:39.795031 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 21:11:39.795035 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 21:11:39.795049 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 21:11:39.795052 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 21:11:39.795096 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0924 21:11:39.795180 (+ 84us) server_negotiation.cc:299] Negotiation successful
0924 21:11:39.795204 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260924 21:11:39.795460 19423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:11:39.791300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33511 (local address 127.0.98.3:46831)
0924 21:11:39.791458 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0924 21:11:39.791490 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0924 21:11:39.791892 (+ 402us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 21:11:39.792128 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 21:11:39.792131 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 21:11:39.792139 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0924 21:11:39.792539 (+ 400us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:11:39.792543 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:11:39.793190 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:11:39.793191 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:11:39.794003 (+ 812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:11:39.794005 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:11:39.794366 (+ 361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:11:39.794368 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:11:39.794476 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:11:39.794910 (+ 434us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 21:11:39.794910 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0924 21:11:39.794932 (+ 22us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 21:11:39.794950 (+ 18us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 21:11:39.795107 (+ 157us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 21:11:39.795110 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 21:11:39.795112 (+ 2us) client_negotiation.cc:769] Sending connection context
0924 21:11:39.795181 (+ 69us) client_negotiation.cc:240] Negotiation successful
0924 21:11:39.795262 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":87,"threads_started":1}
I20260924 21:11:39.795843 19421 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33511
I20260924 21:11:39.795912 19421 heartbeater.cc:464] Registering TS with master...
I20260924 21:11:39.796098 19421 heartbeater.cc:511] Master 127.0.98.62:33511 requested a full tablet report, sending...
I20260924 21:11:39.796402 18971 ts_manager.cc:194] Registered new tserver with Master: 9f90a26f3d364113a853649458ac521d (127.0.98.3:44255)
I20260924 21:11:39.796727 18971 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:46831
I20260924 21:11:39.797928 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAzNNjS/build/release/bin/kudu as pid 19293
I20260924 21:11:39.797996 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790284076800039-392-0/minicluster-data/ts-2/wal/instance
I20260924 21:11:39.798120 392 external_mini_cluster.cc:1428] Setting key dd8088d50a57839716db692f00ffb2c5
I20260924 21:11:39.798933 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 21:11:39.802317 19011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:11:39.800114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54996 (local address 127.0.98.62:33511)
0924 21:11:39.800151 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0924 21:11:39.800153 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 21:11:39.800282 (+ 129us) server_negotiation.cc:407] Connection header received
0924 21:11:39.800297 (+ 15us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:11:39.800298 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:11:39.800304 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:11:39.800336 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:11:39.800449 (+ 113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:11:39.800815 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:11:39.801382 (+ 567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:11:39.801525 (+ 143us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:11:39.801565 (+ 40us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:11:39.802075 (+ 510us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 21:11:39.802076 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 21:11:39.802076 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 21:11:39.802079 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0924 21:11:39.802089 (+ 10us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 21:11:39.802092 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 21:11:39.802153 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0924 21:11:39.802212 (+ 59us) server_negotiation.cc:299] Negotiation successful
0924 21:11:39.802238 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260924 21:11:39.803519 18971 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54996:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 21:11:39.804821 18971 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 21:11:39.810335 19430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:11:39.806616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36899 (local address 127.0.0.1:46224)
0924 21:11:39.806831 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0924 21:11:39.806840 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 21:11:39.806875 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 21:11:39.807134 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 21:11:39.807135 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 21:11:39.807137 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 21:11:39.807164 (+ 27us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:11:39.807165 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:11:39.808303 (+ 1138us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:11:39.808304 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:11:39.809345 (+ 1041us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:11:39.809347 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:11:39.810095 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:11:39.810096 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:11:39.810162 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:11:39.810172 (+ 10us) client_negotiation.cc:769] Sending connection context
0924 21:11:39.810234 (+ 62us) client_negotiation.cc:240] Negotiation successful
0924 21:11:39.810278 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":70,"threads_started":1}
I20260924 21:11:39.810346 19433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:11:39.806849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46224 (local address 127.0.98.1:36899)
0924 21:11:39.806975 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0924 21:11:39.806978 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 21:11:39.806988 (+ 10us) server_negotiation.cc:407] Connection header received
0924 21:11:39.807024 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:11:39.807025 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:11:39.807040 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:11:39.807141 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 21:11:39.807206 (+ 65us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:11:39.808206 (+ 1000us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:11:39.809430 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:11:39.810020 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:11:39.810095 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:11:39.810171 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0924 21:11:39.810247 (+ 76us) server_negotiation.cc:299] Negotiation successful
0924 21:11:39.810281 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94,"thread_start_us":68,"threads_started":1}
I20260924 21:11:39.810534 19434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:11:39.807002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41104 (local address 127.0.98.2:43025)
0924 21:11:39.807310 (+ 308us) server_negotiation.cc:206] Beginning negotiation
0924 21:11:39.807314 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 21:11:39.807321 (+ 7us) server_negotiation.cc:407] Connection header received
0924 21:11:39.807353 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:11:39.807355 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:11:39.807368 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:11:39.807438 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 21:11:39.807675 (+ 237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:11:39.808573 (+ 898us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:11:39.809493 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:11:39.810098 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:11:39.810146 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:11:39.810214 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0924 21:11:39.810429 (+ 215us) server_negotiation.cc:299] Negotiation successful
0924 21:11:39.810461 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":271,"thread_start_us":69,"threads_started":1}
I20260924 21:11:39.810659 19431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:11:39.806646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49426 (local address 127.0.98.3:44255)
0924 21:11:39.806763 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0924 21:11:39.806766 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 21:11:39.806774 (+ 8us) server_negotiation.cc:407] Connection header received
0924 21:11:39.806878 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:11:39.806880 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:11:39.806893 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:11:39.806962 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 21:11:39.807068 (+ 106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:11:39.807980 (+ 912us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:11:39.809363 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:11:39.810083 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:11:39.810146 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:11:39.810213 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0924 21:11:39.810513 (+ 300us) server_negotiation.cc:299] Negotiation successful
0924 21:11:39.810546 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89,"thread_start_us":60,"threads_started":1}
I20260924 21:11:39.810462 19429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:11:39.806578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44255 (local address 127.0.0.1:49426)
0924 21:11:39.806679 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0924 21:11:39.806711 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0924 21:11:39.806766 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 21:11:39.806966 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 21:11:39.806968 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 21:11:39.806969 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 21:11:39.807020 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:11:39.807021 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:11:39.808053 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:11:39.808054 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:11:39.809272 (+ 1218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:11:39.809275 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:11:39.810246 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:11:39.810247 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:11:39.810300 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:11:39.810308 (+ 8us) client_negotiation.cc:769] Sending connection context
0924 21:11:39.810351 (+ 43us) client_negotiation.cc:240] Negotiation successful
0924 21:11:39.810394 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":79,"mutex_wait_us":2,"thread_start_us":58,"threads_started":1}
I20260924 21:11:39.810811 19432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:11:39.806750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43025 (local address 127.0.0.1:41104)
0924 21:11:39.807155 (+ 405us) negotiation.cc:107] Waiting for socket to connect
0924 21:11:39.807164 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 21:11:39.807206 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 21:11:39.807511 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 21:11:39.807512 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 21:11:39.807514 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 21:11:39.807550 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:11:39.807551 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:11:39.808658 (+ 1107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:11:39.808659 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:11:39.809391 (+ 732us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:11:39.809393 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:11:39.810247 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:11:39.810247 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:11:39.810300 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:11:39.810310 (+ 10us) client_negotiation.cc:769] Sending connection context
0924 21:11:39.810351 (+ 41us) client_negotiation.cc:240] Negotiation successful
0924 21:11:39.810394 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":390,"mutex_wait_us":8,"thread_start_us":50,"threads_started":1}
I20260924 21:11:39.811182 19093 tablet_service.cc:1467] Processing CreateTablet for tablet c83e3138b413450ea6e22c931d876de0 (DEFAULT_TABLE table=test-table [id=900cf3a1adfd4905b308c3ccb928f1a1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 21:11:39.811225 19355 tablet_service.cc:1467] Processing CreateTablet for tablet c83e3138b413450ea6e22c931d876de0 (DEFAULT_TABLE table=test-table [id=900cf3a1adfd4905b308c3ccb928f1a1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 21:11:39.811326 19224 tablet_service.cc:1467] Processing CreateTablet for tablet c83e3138b413450ea6e22c931d876de0 (DEFAULT_TABLE table=test-table [id=900cf3a1adfd4905b308c3ccb928f1a1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 21:11:39.811470 19093 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c83e3138b413450ea6e22c931d876de0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 21:11:39.811515 19355 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c83e3138b413450ea6e22c931d876de0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 21:11:39.811575 19224 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c83e3138b413450ea6e22c931d876de0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 21:11:39.813766 19435 tablet_bootstrap.cc:492] T c83e3138b413450ea6e22c931d876de0 P ea57f4697be641f19f5d643b4e13c6b2: Bootstrap starting.
I20260924 21:11:39.814316 19437 tablet_bootstrap.cc:492] T c83e3138b413450ea6e22c931d876de0 P 9f90a26f3d364113a853649458ac521d: Bootstrap starting.
I20260924 21:11:39.814486 19435 tablet_bootstrap.cc:654] T c83e3138b413450ea6e22c931d876de0 P ea57f4697be641f19f5d643b4e13c6b2: Neither blocks nor log segments found. Creating new log.
I20260924 21:11:39.814786 19435 log.cc:824] T c83e3138b413450ea6e22c931d876de0 P ea57f4697be641f19f5d643b4e13c6b2: Log is configured to *not* fsync() on all Append() calls
I20260924 21:11:39.814862 19437 tablet_bootstrap.cc:654] T c83e3138b413450ea6e22c931d876de0 P 9f90a26f3d364113a853649458ac521d: Neither blocks nor log segments found. Creating new log.
I20260924 21:11:39.815160 19437 log.cc:824] T c83e3138b413450ea6e22c931d876de0 P 9f90a26f3d364113a853649458ac521d: Log is configured to *not* fsync() on all Append() calls
I20260924 21:11:39.815426 19436 tablet_bootstrap.cc:492] T c83e3138b413450ea6e22c931d876de0 P a0c6df6d27fd49c9a14ebfeb586d7bc1: Bootstrap starting.
I20260924 21:11:39.815521 19435 tablet_bootstrap.cc:492] T c83e3138b413450ea6e22c931d876de0 P ea57f4697be641f19f5d643b4e13c6b2: No bootstrap required, opened a new log
I20260924 21:11:39.815593 19435 ts_tablet_manager.cc:1397] T c83e3138b413450ea6e22c931d876de0 P ea57f4697be641f19f5d643b4e13c6b2: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 21:11:39.815765 19437 tablet_bootstrap.cc:492] T c83e3138b413450ea6e22c931d876de0 P 9f90a26f3d364113a853649458ac521d: No bootstrap required, opened a new log
I20260924 21:11:39.815832 19437 ts_tablet_manager.cc:1397] T c83e3138b413450ea6e22c931d876de0 P 9f90a26f3d364113a853649458ac521d: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 21:11:39.815968 19436 tablet_bootstrap.cc:654] T c83e3138b413450ea6e22c931d876de0 P a0c6df6d27fd49c9a14ebfeb586d7bc1: Neither blocks nor log segments found. Creating new log.
I20260924 21:11:39.816284 19436 log.cc:824] T c83e3138b413450ea6e22c931d876de0 P a0c6df6d27fd49c9a14ebfeb586d7bc1: Log is configured to *not* fsync() on all Append() calls
I20260924 21:11:39.816936 19436 tablet_bootstrap.cc:492] T c83e3138b413450ea6e22c931d876de0 P a0c6df6d27fd49c9a14ebfeb586d7bc1: No bootstrap required, opened a new log
I20260924 21:11:39.816994 19436 ts_tablet_manager.cc:1397] T c83e3138b413450ea6e22c931d876de0 P a0c6df6d27fd49c9a14ebfeb586d7bc1: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 21:11:39.817582 19437 raft_consensus.cc:348] T c83e3138b413450ea6e22c931d876de0 P 9f90a26f3d364113a853649458ac521d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f90a26f3d364113a853649458ac521d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44255 } } peers { permanent_uuid: "a0c6df6d27fd49c9a14ebfeb586d7bc1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43025 } } peers { permanent_uuid: "ea57f4697be641f19f5d643b4e13c6b2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36899 } }
I20260924 21:11:39.817709 19437 raft_consensus.cc:374] T c83e3138b413450ea6e22c931d876de0 P 9f90a26f3d364113a853649458ac521d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 21:11:39.817744 19437 raft_consensus.cc:729] T c83e3138b413450ea6e22c931d876de0 P 9f90a26f3d364113a853649458ac521d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f90a26f3d364113a853649458ac521d, State: Initialized, Role: FOLLOWER
I20260924 21:11:39.817764 19435 raft_consensus.cc:348] T c83e3138b413450ea6e22c931d876de0 P ea57f4697be641f19f5d643b4e13c6b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f90a26f3d364113a853649458ac521d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44255 } } peers { permanent_uuid: "a0c6df6d27fd49c9a14ebfeb586d7bc1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43025 } } peers { permanent_uuid: "ea57f4697be641f19f5d643b4e13c6b2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36899 } }
I20260924 21:11:39.817911 19435 raft_consensus.cc:374] T c83e3138b413450ea6e22c931d876de0 P ea57f4697be641f19f5d643b4e13c6b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 21:11:39.817870 19437 consensus_queue.cc:260] T c83e3138b413450ea6e22c931d876de0 P 9f90a26f3d364113a853649458ac521d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f90a26f3d364113a853649458ac521d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44255 } } peers { permanent_uuid: "a0c6df6d27fd49c9a14ebfeb586d7bc1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43025 } } peers { permanent_uuid: "ea57f4697be641f19f5d643b4e13c6b2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36899 } }
I20260924 21:11:39.817940 19435 raft_consensus.cc:729] T c83e3138b413450ea6e22c931d876de0 P ea57f4697be641f19f5d643b4e13c6b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea57f4697be641f19f5d643b4e13c6b2, State: Initialized, Role: FOLLOWER
I20260924 21:11:39.818044 19435 consensus_queue.cc:260] T c83e3138b413450ea6e22c931d876de0 P ea57f4697be641f19f5d643b4e13c6b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f90a26f3d364113a853649458ac521d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44255 } } peers { permanent_uuid: "a0c6df6d27fd49c9a14ebfeb586d7bc1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43025 } } peers { permanent_uuid: "ea57f4697be641f19f5d643b4e13c6b2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36899 } }
I20260924 21:11:39.818120 19437 ts_tablet_manager.cc:1428] T c83e3138b413450ea6e22c931d876de0 P 9f90a26f3d364113a853649458ac521d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 21:11:39.818225 19421 heartbeater.cc:503] Master 127.0.98.62:33511 was elected leader, sending a full tablet report...
I20260924 21:11:39.818547 19435 ts_tablet_manager.cc:1428] T c83e3138b413450ea6e22c931d876de0 P ea57f4697be641f19f5d643b4e13c6b2: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 21:11:39.818699 19159 heartbeater.cc:503] Master 127.0.98.62:33511 was elected leader, sending a full tablet report...
I20260924 21:11:39.819350 19436 raft_consensus.cc:348] T c83e3138b413450ea6e22c931d876de0 P a0c6df6d27fd49c9a14ebfeb586d7bc1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f90a26f3d364113a853649458ac521d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44255 } } peers { permanent_uuid: "a0c6df6d27fd49c9a14ebfeb586d7bc1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43025 } } peers { permanent_uuid: "ea57f4697be641f19f5d643b4e13c6b2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36899 } }
I20260924 21:11:39.819545 19436 raft_consensus.cc:374] T c83e3138b413450ea6e22c931d876de0 P a0c6df6d27fd49c9a14ebfeb586d7bc1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 21:11:39.819576 19436 raft_consensus.cc:729] T c83e3138b413450ea6e22c931d876de0 P a0c6df6d27fd49c9a14ebfeb586d7bc1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0c6df6d27fd49c9a14ebfeb586d7bc1, State: Initialized, Role: FOLLOWER
I20260924 21:11:39.819702 19436 consensus_queue.cc:260] T c83e3138b413450ea6e22c931d876de0 P a0c6df6d27fd49c9a14ebfeb586d7bc1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f90a26f3d364113a853649458ac521d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44255 } } peers { permanent_uuid: "a0c6df6d27fd49c9a14ebfeb586d7bc1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43025 } } peers { permanent_uuid: "ea57f4697be641f19f5d643b4e13c6b2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36899 } }
I20260924 21:11:39.819916 19436 ts_tablet_manager.cc:1428] T c83e3138b413450ea6e22c931d876de0 P a0c6df6d27fd49c9a14ebfeb586d7bc1: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 21:11:39.819955 19290 heartbeater.cc:503] Master 127.0.98.62:33511 was elected leader, sending a full tablet report...
W20260924 21:11:39.925964 19291 tablet.cc:2369] T c83e3138b413450ea6e22c931d876de0 P a0c6df6d27fd49c9a14ebfeb586d7bc1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 21:11:39.970669 19442 raft_consensus.cc:482] T c83e3138b413450ea6e22c931d876de0 P ea57f4697be641f19f5d643b4e13c6b2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 21:11:39.970819 19442 raft_consensus.cc:504] T c83e3138b413450ea6e22c931d876de0 P ea57f4697be641f19f5d643b4e13c6b2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f90a26f3d364113a853649458ac521d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44255 } } peers { permanent_uuid: "a0c6df6d27fd49c9a14ebfeb586d7bc1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43025 } } peers { permanent_uuid: "ea57f4697be641f19f5d643b4e13c6b2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36899 } }
I20260924 21:11:39.971107 19442 leader_election.cc:288] T c83e3138b413450ea6e22c931d876de0 P ea57f4697be641f19f5d643b4e13c6b2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9f90a26f3d364113a853649458ac521d (127.0.98.3:44255), a0c6df6d27fd49c9a14ebfeb586d7bc1 (127.0.98.2:43025)
I20260924 21:11:39.973677 19431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:11:39.971351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:48479 (local address 127.0.98.3:44255)
0924 21:11:39.971395 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0924 21:11:39.971397 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 21:11:39.971406 (+ 9us) server_negotiation.cc:407] Connection header received
0924 21:11:39.971432 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:11:39.971433 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:11:39.971443 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:11:39.971500 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 21:11:39.971681 (+ 181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:11:39.972163 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:11:39.973010 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:11:39.973331 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:11:39.973404 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:11:39.973413 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0924 21:11:39.973567 (+ 154us) server_negotiation.cc:299] Negotiation successful
0924 21:11:39.973602 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260924 21:11:39.973677 19161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:11:39.971262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44255 (local address 127.0.98.1:48479)
0924 21:11:39.971310 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0924 21:11:39.971320 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 21:11:39.971359 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 21:11:39.971514 (+ 155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 21:11:39.971515 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 21:11:39.971516 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 21:11:39.971602 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:11:39.971603 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:11:39.972236 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:11:39.972237 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:11:39.972925 (+ 688us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:11:39.972927 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:11:39.973405 (+ 478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:11:39.973406 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:11:39.973475 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:11:39.973478 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 21:11:39.973562 (+ 84us) client_negotiation.cc:240] Negotiation successful
0924 21:11:39.973601 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28,"mutex_wait_us":3}
I20260924 21:11:39.973838 19434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:11:39.971423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:38301 (local address 127.0.98.2:43025)
0924 21:11:39.971477 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0924 21:11:39.971478 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 21:11:39.971562 (+ 84us) server_negotiation.cc:407] Connection header received
0924 21:11:39.971646 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:11:39.971647 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:11:39.971655 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:11:39.971690 (+ 35us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 21:11:39.971840 (+ 150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:11:39.972234 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:11:39.973102 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:11:39.973430 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:11:39.973521 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:11:39.973530 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0924 21:11:39.973697 (+ 167us) server_negotiation.cc:299] Negotiation successful
0924 21:11:39.973731 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260924 21:11:39.973948 19444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:11:39.971363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43025 (local address 127.0.98.1:38301)
0924 21:11:39.971492 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0924 21:11:39.971499 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0924 21:11:39.971564 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 21:11:39.971700 (+ 136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 21:11:39.971700 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 21:11:39.971701 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 21:11:39.971780 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:11:39.971781 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:11:39.972307 (+ 526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:11:39.972308 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:11:39.972997 (+ 689us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:11:39.972999 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:11:39.973524 (+ 525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:11:39.973525 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:11:39.973610 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:11:39.973613 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 21:11:39.973681 (+ 68us) client_negotiation.cc:240] Negotiation successful
0924 21:11:39.973868 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":86,"threads_started":1}
I20260924 21:11:39.974224 19375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c83e3138b413450ea6e22c931d876de0" candidate_uuid: "ea57f4697be641f19f5d643b4e13c6b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f90a26f3d364113a853649458ac521d" is_pre_election: true
I20260924 21:11:39.974236 19244 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c83e3138b413450ea6e22c931d876de0" candidate_uuid: "ea57f4697be641f19f5d643b4e13c6b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0c6df6d27fd49c9a14ebfeb586d7bc1" is_pre_election: true
I20260924 21:11:39.974396 19375 raft_consensus.cc:2454] T c83e3138b413450ea6e22c931d876de0 P 9f90a26f3d364113a853649458ac521d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ea57f4697be641f19f5d643b4e13c6b2 in term 0.
I20260924 21:11:39.974396 19244 raft_consensus.cc:2454] T c83e3138b413450ea6e22c931d876de0 P a0c6df6d27fd49c9a14ebfeb586d7bc1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ea57f4697be641f19f5d643b4e13c6b2 in term 0.
I20260924 21:11:39.974612 19049 leader_election.cc:302] T c83e3138b413450ea6e22c931d876de0 P ea57f4697be641f19f5d643b4e13c6b2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a0c6df6d27fd49c9a14ebfeb586d7bc1, ea57f4697be641f19f5d643b4e13c6b2; no voters:
I20260924 21:11:39.974725 19442 raft_consensus.cc:2781] T c83e3138b413450ea6e22c931d876de0 P ea57f4697be641f19f5d643b4e13c6b2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 21:11:39.974818 19442 raft_consensus.cc:482] T c83e3138b413450ea6e22c931d876de0 P ea57f4697be641f19f5d643b4e13c6b2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 21:11:39.974850 19442 raft_consensus.cc:3037] T c83e3138b413450ea6e22c931d876de0 P ea57f4697be641f19f5d643b4e13c6b2 [term 0 FOLLOWER]: Advancing to term 1
I20260924 21:11:39.975565 19442 raft_consensus.cc:504] T c83e3138b413450ea6e22c931d876de0 P ea57f4697be641f19f5d643b4e13c6b2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f90a26f3d364113a853649458ac521d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44255 } } peers { permanent_uuid: "a0c6df6d27fd49c9a14ebfeb586d7bc1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43025 } } peers { permanent_uuid: "ea57f4697be641f19f5d643b4e13c6b2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36899 } }
I20260924 21:11:39.975692 19442 leader_election.cc:288] T c83e3138b413450ea6e22c931d876de0 P ea57f4697be641f19f5d643b4e13c6b2 [CANDIDATE]: Term 1 election: Requested vote from peers 9f90a26f3d364113a853649458ac521d (127.0.98.3:44255), a0c6df6d27fd49c9a14ebfeb586d7bc1 (127.0.98.2:43025)
I20260924 21:11:39.975867 19375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c83e3138b413450ea6e22c931d876de0" candidate_uuid: "ea57f4697be641f19f5d643b4e13c6b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f90a26f3d364113a853649458ac521d"
I20260924 21:11:39.975872 19244 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c83e3138b413450ea6e22c931d876de0" candidate_uuid: "ea57f4697be641f19f5d643b4e13c6b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0c6df6d27fd49c9a14ebfeb586d7bc1"
I20260924 21:11:39.975943 19375 raft_consensus.cc:3037] T c83e3138b413450ea6e22c931d876de0 P 9f90a26f3d364113a853649458ac521d [term 0 FOLLOWER]: Advancing to term 1
I20260924 21:11:39.975943 19244 raft_consensus.cc:3037] T c83e3138b413450ea6e22c931d876de0 P a0c6df6d27fd49c9a14ebfeb586d7bc1 [term 0 FOLLOWER]: Advancing to term 1
I20260924 21:11:39.976801 19244 raft_consensus.cc:2454] T c83e3138b413450ea6e22c931d876de0 P a0c6df6d27fd49c9a14ebfeb586d7bc1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ea57f4697be641f19f5d643b4e13c6b2 in term 1.
I20260924 21:11:39.976801 19375 raft_consensus.cc:2454] T c83e3138b413450ea6e22c931d876de0 P 9f90a26f3d364113a853649458ac521d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ea57f4697be641f19f5d643b4e13c6b2 in term 1.
I20260924 21:11:39.976951 19049 leader_election.cc:302] T c83e3138b413450ea6e22c931d876de0 P ea57f4697be641f19f5d643b4e13c6b2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a0c6df6d27fd49c9a14ebfeb586d7bc1, ea57f4697be641f19f5d643b4e13c6b2; no voters:
I20260924 21:11:39.977042 19442 raft_consensus.cc:2781] T c83e3138b413450ea6e22c931d876de0 P ea57f4697be641f19f5d643b4e13c6b2 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 21:11:39.977257 19442 raft_consensus.cc:686] T c83e3138b413450ea6e22c931d876de0 P ea57f4697be641f19f5d643b4e13c6b2 [term 1 LEADER]: Becoming Leader. State: Replica: ea57f4697be641f19f5d643b4e13c6b2, State: Running, Role: LEADER
I20260924 21:11:39.977360 19442 consensus_queue.cc:237] T c83e3138b413450ea6e22c931d876de0 P ea57f4697be641f19f5d643b4e13c6b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f90a26f3d364113a853649458ac521d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44255 } } peers { permanent_uuid: "a0c6df6d27fd49c9a14ebfeb586d7bc1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43025 } } peers { permanent_uuid: "ea57f4697be641f19f5d643b4e13c6b2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36899 } }
I20260924 21:11:39.978219 18970 catalog_manager.cc:5495] T c83e3138b413450ea6e22c931d876de0 P ea57f4697be641f19f5d643b4e13c6b2 reported cstate change: term changed from 0 to 1, leader changed from <none> to ea57f4697be641f19f5d643b4e13c6b2 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "ea57f4697be641f19f5d643b4e13c6b2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f90a26f3d364113a853649458ac521d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44255 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a0c6df6d27fd49c9a14ebfeb586d7bc1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43025 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ea57f4697be641f19f5d643b4e13c6b2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36899 } health_report { overall_health: HEALTHY } } }
I20260924 21:11:39.989306 19433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:11:39.987217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46230 (local address 127.0.98.1:36899)
0924 21:11:39.987255 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0924 21:11:39.987256 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 21:11:39.987263 (+ 7us) server_negotiation.cc:407] Connection header received
0924 21:11:39.987333 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:11:39.987334 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:11:39.987347 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:11:39.987398 (+ 51us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 21:11:39.987501 (+ 103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:11:39.987880 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:11:39.988621 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:11:39.988786 (+ 165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:11:39.988832 (+ 46us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:11:39.988964 (+ 132us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 21:11:39.989040 (+ 76us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 21:11:39.989130 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0924 21:11:39.989187 (+ 57us) server_negotiation.cc:299] Negotiation successful
0924 21:11:39.989196 (+ 9us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260924 21:11:39.991040 19244 raft_consensus.cc:1261] T c83e3138b413450ea6e22c931d876de0 P a0c6df6d27fd49c9a14ebfeb586d7bc1 [term 1 FOLLOWER]: Refusing update from remote peer ea57f4697be641f19f5d643b4e13c6b2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 21:11:39.991092 19375 raft_consensus.cc:1261] T c83e3138b413450ea6e22c931d876de0 P 9f90a26f3d364113a853649458ac521d [term 1 FOLLOWER]: Refusing update from remote peer ea57f4697be641f19f5d643b4e13c6b2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 21:11:39.991333 19442 consensus_queue.cc:1035] T c83e3138b413450ea6e22c931d876de0 P ea57f4697be641f19f5d643b4e13c6b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0c6df6d27fd49c9a14ebfeb586d7bc1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 21:11:39.991462 19445 consensus_queue.cc:1035] T c83e3138b413450ea6e22c931d876de0 P ea57f4697be641f19f5d643b4e13c6b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f90a26f3d364113a853649458ac521d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44255 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 21:11:39.993829 19448 mvcc.cc:205] Tried to move back new op lower bound from 7333004492760784896 to 7333004492707848192. Current Snapshot: MvccSnapshot[applied={T|T < 7333004492760784896}]
I20260924 21:11:39.995754 18971 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54996:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 21:11:39.995839 18971 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54996:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 21:11:39.996734 18971 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b1dc4d7e0125486296c0bf6497347786: Sending DeleteTablet for 3 replicas of tablet c83e3138b413450ea6e22c931d876de0
I20260924 21:11:39.997149 19093 tablet_service.cc:1514] Processing DeleteTablet for tablet c83e3138b413450ea6e22c931d876de0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 21:11:39 UTC) from {username='slave'} at 127.0.0.1:46224
I20260924 21:11:39.997359 19355 tablet_service.cc:1514] Processing DeleteTablet for tablet c83e3138b413450ea6e22c931d876de0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 21:11:39 UTC) from {username='slave'} at 127.0.0.1:49426
I20260924 21:11:39.997412 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAzNNjS/build/release/bin/kudu with pid 19031
I20260924 21:11:39.998241 19224 tablet_service.cc:1514] Processing DeleteTablet for tablet c83e3138b413450ea6e22c931d876de0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 21:11:39 UTC) from {username='slave'} at 127.0.0.1:41104
I20260924 21:11:39.998454 19460 tablet_replica.cc:330] T c83e3138b413450ea6e22c931d876de0 P a0c6df6d27fd49c9a14ebfeb586d7bc1: stopping tablet replica
I20260924 21:11:39.998554 19460 raft_consensus.cc:2229] T c83e3138b413450ea6e22c931d876de0 P a0c6df6d27fd49c9a14ebfeb586d7bc1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 21:11:39.998588 19459 tablet_replica.cc:330] T c83e3138b413450ea6e22c931d876de0 P 9f90a26f3d364113a853649458ac521d: stopping tablet replica
I20260924 21:11:39.998656 19460 raft_consensus.cc:2258] T c83e3138b413450ea6e22c931d876de0 P a0c6df6d27fd49c9a14ebfeb586d7bc1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 21:11:39.998765 19459 raft_consensus.cc:2229] T c83e3138b413450ea6e22c931d876de0 P 9f90a26f3d364113a853649458ac521d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 21:11:39.998900 19459 raft_consensus.cc:2258] T c83e3138b413450ea6e22c931d876de0 P 9f90a26f3d364113a853649458ac521d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 21:11:39.999046 19460 ts_tablet_manager.cc:1905] T c83e3138b413450ea6e22c931d876de0 P a0c6df6d27fd49c9a14ebfeb586d7bc1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 21:11:39.999403 19459 ts_tablet_manager.cc:1905] T c83e3138b413450ea6e22c931d876de0 P 9f90a26f3d364113a853649458ac521d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 21:11:40.000744 19460 ts_tablet_manager.cc:1918] T c83e3138b413450ea6e22c931d876de0 P a0c6df6d27fd49c9a14ebfeb586d7bc1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 21:11:40.000818 19460 log.cc:1196] T c83e3138b413450ea6e22c931d876de0 P a0c6df6d27fd49c9a14ebfeb586d7bc1: Deleting WAL directory at /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790284076800039-392-0/minicluster-data/ts-1/wal/wals/c83e3138b413450ea6e22c931d876de0
I20260924 21:11:40.001077 19460 ts_tablet_manager.cc:1939] T c83e3138b413450ea6e22c931d876de0 P a0c6df6d27fd49c9a14ebfeb586d7bc1: Deleting consensus metadata
I20260924 21:11:40.001299 19459 ts_tablet_manager.cc:1918] T c83e3138b413450ea6e22c931d876de0 P 9f90a26f3d364113a853649458ac521d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 21:11:40.001384 19459 log.cc:1196] T c83e3138b413450ea6e22c931d876de0 P 9f90a26f3d364113a853649458ac521d: Deleting WAL directory at /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790284076800039-392-0/minicluster-data/ts-2/wal/wals/c83e3138b413450ea6e22c931d876de0
I20260924 21:11:40.001446 18958 catalog_manager.cc:4841] TS a0c6df6d27fd49c9a14ebfeb586d7bc1 (127.0.98.2:43025): tablet c83e3138b413450ea6e22c931d876de0 (table test-table [id=900cf3a1adfd4905b308c3ccb928f1a1]) successfully deleted
I20260924 21:11:40.001606 19459 ts_tablet_manager.cc:1939] T c83e3138b413450ea6e22c931d876de0 P 9f90a26f3d364113a853649458ac521d: Deleting consensus metadata
I20260924 21:11:40.001960 18958 catalog_manager.cc:4841] TS 9f90a26f3d364113a853649458ac521d (127.0.98.3:44255): tablet c83e3138b413450ea6e22c931d876de0 (table test-table [id=900cf3a1adfd4905b308c3ccb928f1a1]) successfully deleted
W20260924 21:11:40.003234 18955 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:36899 (error 108)
I20260924 21:11:40.003641 19429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:11:40.003525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36899 (local address 127.0.0.1:46244)
0924 21:11:40.003578 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0924 21:11:40.003614 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:36899: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260924 21:11:40.003698 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAzNNjS/build/release/bin/kudu with pid 19162
W20260924 21:11:40.003750 18955 catalog_manager.cc:4584] TS ea57f4697be641f19f5d643b4e13c6b2 (127.0.98.1:36899): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c83e3138b413450ea6e22c931d876de0: Network error: Client connection negotiation failed: client connection to 127.0.98.1:36899: connect: Connection refused (error 111)
I20260924 21:11:40.007850 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAzNNjS/build/release/bin/kudu with pid 19293
I20260924 21:11:40.012212 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAzNNjS/build/release/bin/kudu with pid 18940
2026-09-24T21:11:40Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (730 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790284076800039-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 21:11:40 dist-test-slave-c7tk krb5kdc[19464](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 21:11:40 dist-test-slave-c7tk krb5kdc[19464](info): set up 2 sockets
Sep 24 21:11:40 dist-test-slave-c7tk krb5kdc[19464](info): commencing operation
krb5kdc: starting...
W20260924 21:11:40.953442 392 mini_kdc.cc:121] Time spent starting KDC: real 0.923s user 0.001s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790284076800039-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790284076800039-392-0/krb5kdc/test-user.keytab.
Sep 24 21:11:41 dist-test-slave-c7tk krb5kdc[19464](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284301, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T21:11:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T21:11:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790284076800039-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790284076800039-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 21:11:41.056008 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAzNNjS/build/release/bin/kudu
/tmp/dist-test-taskAzNNjS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790284076800039-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790284076800039-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790284076800039-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790284076800039-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44011
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790284076800039-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34603
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44011
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790284076800039-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790284076800039-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790284076800039-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 21:11:41.124375 19477 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 21:11:41.124568 19477 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 21:11:41.124601 19477 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 21:11:41.125550 19477 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 21:11:41.125615 19477 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 21:11:41.125635 19477 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 21:11:41.125659 19477 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 21:11:41.125685 19477 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 21:11:41.127019 19477 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34603
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790284076800039-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790284076800039-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44011
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790284076800039-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44011
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790284076800039-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19477
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790284076800039-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 19:49:06 UTC on ccycloud.quasar-vhqjoc.root.comops.site
build id 4670864
I20260924 21:11:41.127259 19477 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 21:11:41.127463 19477 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 21:11:41.131798 19477 server_base.cc:1034] running on GCE node
W20260924 21:11:41.131763 19485 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 21:11:41.131759 19483 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 21:11:41.131785 19482 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 21:11:41.132128 19477 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 21:11:41.132395 19477 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 21:11:41.133533 19477 hybrid_clock.cc:648] HybridClock initialized: now 1790284301133528 us; error 31 us; skew 500 ppm
Sep 24 21:11:41 dist-test-slave-c7tk krb5kdc[19464](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284301, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 21:11:41.135584 19477 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 21:11:41.135991 19477 webserver.cc:460] Webserver started at http://127.0.98.62:34719/ using document root <none> and password file <none>
I20260924 21:11:41.136190 19477 fs_manager.cc:359] Metadata directory not provided
I20260924 21:11:41.136250 19477 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 21:11:41.136361 19477 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 21:11:41.137194 19477 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790284076800039-392-0/minicluster-data/master-0/data/instance:
uuid: "54b15ed90a9f4be5a6585de359f0af12"
format_stamp: "Formatted at 2026-09-24 21:11:41 on dist-test-slave-c7tk"
server_key: "fbeb5b9cf5874ee5fd733363b02e7e74"
server_key_iv: "e0e4be86a45687985e17be08bb27341f"
server_key_version: "encryptionkey@0"
I20260924 21:11:41.137490 19477 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790284076800039-392-0/minicluster-data/master-0/wal/instance:
uuid: "54b15ed90a9f4be5a6585de359f0af12"
format_stamp: "Formatted at 2026-09-24 21:11:41 on dist-test-slave-c7tk"
server_key: "fbeb5b9cf5874ee5fd733363b02e7e74"
server_key_iv: "e0e4be86a45687985e17be08bb27341f"
server_key_version: "encryptionkey@0"
I20260924 21:11:41.138710 19477 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20260924 21:11:41.139380 19492 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 21:11:41.139539 19477 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 21:11:41.139643 19477 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790284076800039-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790284076800039-392-0/minicluster-data/master-0/wal
uuid: "54b15ed90a9f4be5a6585de359f0af12"
format_stamp: "Formatted at 2026-09-24 21:11:41 on dist-test-slave-c7tk"
server_key: "fbeb5b9cf5874ee5fd733363b02e7e74"
server_key_iv: "e0e4be86a45687985e17be08bb27341f"
server_key_version: "encryptionkey@0"
I20260924 21:11:41.139715 19477 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790284076800039-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790284076800039-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790284076800039-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 21:11:41.162968 19477 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 21:11:41.164420 19477 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 21:11:41.164568 19477 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 21:11:41.168051 19477 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44011
I20260924 21:11:41.168078 19544 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44011 every 8 connection(s)
I20260924 21:11:41.168347 19477 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790284076800039-392-0/minicluster-data/master-0/data/info.pb
I20260924 21:11:41.169158 19545 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 21:11:41.169577 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAzNNjS/build/release/bin/kudu as pid 19477
I20260924 21:11:41.169704 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790284076800039-392-0/minicluster-data/master-0/wal/instance
I20260924 21:11:41.169895 392 external_mini_cluster.cc:1428] Setting key d1c171b6dfad64cfd75919499a04545e
I20260924 21:11:41.171813 19545 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 54b15ed90a9f4be5a6585de359f0af12: Bootstrap starting.
I20260924 21:11:41.172303 19545 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 54b15ed90a9f4be5a6585de359f0af12: Neither blocks nor log segments found. Creating new log.
I20260924 21:11:41.172572 19545 log.cc:824] T 00000000000000000000000000000000 P 54b15ed90a9f4be5a6585de359f0af12: Log is configured to *not* fsync() on all Append() calls
I20260924 21:11:41.173184 19545 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 54b15ed90a9f4be5a6585de359f0af12: No bootstrap required, opened a new log
I20260924 21:11:41.175338 19545 raft_consensus.cc:348] T 00000000000000000000000000000000 P 54b15ed90a9f4be5a6585de359f0af12 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54b15ed90a9f4be5a6585de359f0af12" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44011 } }
I20260924 21:11:41.175443 19545 raft_consensus.cc:374] T 00000000000000000000000000000000 P 54b15ed90a9f4be5a6585de359f0af12 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 21:11:41.175462 19545 raft_consensus.cc:729] T 00000000000000000000000000000000 P 54b15ed90a9f4be5a6585de359f0af12 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54b15ed90a9f4be5a6585de359f0af12, State: Initialized, Role: FOLLOWER
I20260924 21:11:41.175583 19545 consensus_queue.cc:260] T 00000000000000000000000000000000 P 54b15ed90a9f4be5a6585de359f0af12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54b15ed90a9f4be5a6585de359f0af12" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44011 } }
I20260924 21:11:41.175640 19545 raft_consensus.cc:388] T 00000000000000000000000000000000 P 54b15ed90a9f4be5a6585de359f0af12 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 21:11:41.175681 19545 raft_consensus.cc:482] T 00000000000000000000000000000000 P 54b15ed90a9f4be5a6585de359f0af12 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 21:11:41.175724 19545 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 54b15ed90a9f4be5a6585de359f0af12 [term 0 FOLLOWER]: Advancing to term 1
Sep 24 21:11:41 dist-test-slave-c7tk krb5kdc[19464](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284301, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 21:11:41.176294 19545 raft_consensus.cc:504] T 00000000000000000000000000000000 P 54b15ed90a9f4be5a6585de359f0af12 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54b15ed90a9f4be5a6585de359f0af12" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44011 } }
I20260924 21:11:41.176393 19545 leader_election.cc:302] T 00000000000000000000000000000000 P 54b15ed90a9f4be5a6585de359f0af12 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 54b15ed90a9f4be5a6585de359f0af12; no voters:
I20260924 21:11:41.176532 19545 leader_election.cc:288] T 00000000000000000000000000000000 P 54b15ed90a9f4be5a6585de359f0af12 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 21:11:41.176605 19550 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 54b15ed90a9f4be5a6585de359f0af12 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 21:11:41.176751 19545 sys_catalog.cc:564] T 00000000000000000000000000000000 P 54b15ed90a9f4be5a6585de359f0af12 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 21:11:41.177132 19550 raft_consensus.cc:686] T 00000000000000000000000000000000 P 54b15ed90a9f4be5a6585de359f0af12 [term 1 LEADER]: Becoming Leader. State: Replica: 54b15ed90a9f4be5a6585de359f0af12, State: Running, Role: LEADER
I20260924 21:11:41.177984 19550 consensus_queue.cc:237] T 00000000000000000000000000000000 P 54b15ed90a9f4be5a6585de359f0af12 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54b15ed90a9f4be5a6585de359f0af12" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44011 } }
I20260924 21:11:41.178557 19551 sys_catalog.cc:455] T 00000000000000000000000000000000 P 54b15ed90a9f4be5a6585de359f0af12 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "54b15ed90a9f4be5a6585de359f0af12" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54b15ed90a9f4be5a6585de359f0af12" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44011 } } }
I20260924 21:11:41.178557 19552 sys_catalog.cc:455] T 00000000000000000000000000000000 P 54b15ed90a9f4be5a6585de359f0af12 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 54b15ed90a9f4be5a6585de359f0af12. Latest consensus state: current_term: 1 leader_uuid: "54b15ed90a9f4be5a6585de359f0af12" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54b15ed90a9f4be5a6585de359f0af12" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44011 } } }
I20260924 21:11:41.178782 19551 sys_catalog.cc:458] T 00000000000000000000000000000000 P 54b15ed90a9f4be5a6585de359f0af12 [sys.catalog]: This master's current role is: LEADER
I20260924 21:11:41.178790 19552 sys_catalog.cc:458] T 00000000000000000000000000000000 P 54b15ed90a9f4be5a6585de359f0af12 [sys.catalog]: This master's current role is: LEADER
I20260924 21:11:41.178972 19565 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 21:11:41.179390 19565 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 21:11:41.179643 19548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:11:41.170351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40362 (local address 127.0.98.62:44011)
0924 21:11:41.170493 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0924 21:11:41.170500 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 21:11:41.170531 (+ 31us) server_negotiation.cc:407] Connection header received
0924 21:11:41.170849 (+ 318us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:11:41.170854 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:11:41.170892 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:11:41.170987 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:11:41.171682 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:11:41.172515 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:11:41.173372 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:11:41.173701 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:11:41.173769 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:11:41.176203 (+ 2434us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 21:11:41.176207 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 21:11:41.176207 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 21:11:41.176217 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0924 21:11:41.177891 (+ 1674us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:11:41.178210 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:11:41.178212 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:11:41.178212 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:11:41.178286 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:11:41.178565 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:11:41.178566 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:11:41.178566 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:11:41.178870 (+ 304us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 21:11:41.179009 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0924 21:11:41.179243 (+ 234us) server_negotiation.cc:299] Negotiation successful
0924 21:11:41.179373 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105,"thread_start_us":76,"threads_started":1}
I20260924 21:11:41.180645 19565 catalog_manager.cc:1347] Generated new cluster ID: 161e5bd35de5440388fd94c4a173eac6
I20260924 21:11:41.180680 19565 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 21:11:41.184525 19565 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 21:11:41.185191 19565 catalog_manager.cc:1504] Loading token signing keys...
I20260924 21:11:41.191330 19565 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 54b15ed90a9f4be5a6585de359f0af12: Generated new TSK 0
I20260924 21:11:41.191461 19565 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790284076800039-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790284076800039-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260924 21:11:41.217778 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAzNNjS/build/release/bin/kudu
/tmp/dist-test-taskAzNNjS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790284076800039-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790284076800039-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790284076800039-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790284076800039-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44011
--keytab_file=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790284076800039-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34603
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790284076800039-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790284076800039-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790284076800039-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 21:11:41.290848 19571 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 21:11:41.291059 19571 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 21:11:41.291085 19571 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 21:11:41.292343 19571 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 21:11:41.292387 19571 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 21:11:41.292418 19571 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 21:11:41.293913 19571 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34603
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790284076800039-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790284076800039-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790284076800039-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790284076800039-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44011
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19571
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790284076800039-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 19:49:06 UTC on ccycloud.quasar-vhqjoc.root.comops.site
build id 4670864
I20260924 21:11:41.294090 19571 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 21:11:41.294229 19571 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 21:11:41.298260 19576 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 21:11:41.298280 19577 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 21:11:41.298311 19571 server_base.cc:1034] running on GCE node
W20260924 21:11:41.298264 19579 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 21:11:41.298662 19571 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 21:11:41.298854 19571 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 21:11:41.300006 19571 hybrid_clock.cc:648] HybridClock initialized: now 1790284301299985 us; error 37 us; skew 500 ppm
Sep 24 21:11:41 dist-test-slave-c7tk krb5kdc[19464](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284301, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 21:11:41.302022 19571 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 21:11:41.302392 19571 webserver.cc:460] Webserver started at http://127.0.98.1:43189/ using document root <none> and password file <none>
I20260924 21:11:41.302578 19571 fs_manager.cc:359] Metadata directory not provided
I20260924 21:11:41.302615 19571 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 21:11:41.302766 19571 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 21:11:41.303699 19571 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790284076800039-392-0/minicluster-data/ts-0/data/instance:
uuid: "d23fa4b8d26a4a87bcb9817236b7f1a2"
format_stamp: "Formatted at 2026-09-24 21:11:41 on dist-test-slave-c7tk"
server_key: "0fb159a08b712e8ceb8b428343a64516"
server_key_iv: "031f40f48732a8388be7bee0d0419b9b"
server_key_version: "encryptionkey@0"
I20260924 21:11:41.304042 19571 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790284076800039-392-0/minicluster-data/ts-0/wal/instance:
uuid: "d23fa4b8d26a4a87bcb9817236b7f1a2"
format_stamp: "Formatted at 2026-09-24 21:11:41 on dist-test-slave-c7tk"
server_key: "0fb159a08b712e8ceb8b428343a64516"
server_key_iv: "031f40f48732a8388be7bee0d0419b9b"
server_key_version: "encryptionkey@0"
I20260924 21:11:41.305086 19571 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 21:11:41.305652 19586 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 21:11:41.305825 19571 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 21:11:41.305917 19571 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790284076800039-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790284076800039-392-0/minicluster-data/ts-0/wal
uuid: "d23fa4b8d26a4a87bcb9817236b7f1a2"
format_stamp: "Formatted at 2026-09-24 21:11:41 on dist-test-slave-c7tk"
server_key: "0fb159a08b712e8ceb8b428343a64516"
server_key_iv: "031f40f48732a8388be7bee0d0419b9b"
server_key_version: "encryptionkey@0"
I20260924 21:11:41.305989 19571 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790284076800039-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790284076800039-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790284076800039-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 21:11:41.317335 19571 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 21:11:41.318832 19571 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 21:11:41.318981 19571 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 21:11:41.319177 19571 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 21:11:41.319438 19571 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 21:11:41.319489 19571 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 21:11:41.319535 19571 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 21:11:41.319593 19571 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 21:11:41.324429 19571 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:45995
I20260924 21:11:41.324476 19699 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:45995 every 8 connection(s)
I20260924 21:11:41.324719 19571 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790284076800039-392-0/minicluster-data/ts-0/data/info.pb
Sep 24 21:11:41 dist-test-slave-c7tk krb5kdc[19464](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284301, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 21:11:41.331463 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAzNNjS/build/release/bin/kudu as pid 19571
I20260924 21:11:41.331576 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790284076800039-392-0/minicluster-data/ts-0/wal/instance
I20260924 21:11:41.331784 392 external_mini_cluster.cc:1428] Setting key 259b738aa15b04a6c1a168a9698c6f3c
I20260924 21:11:41.332883 19548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:11:41.325355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:38221 (local address 127.0.98.62:44011)
0924 21:11:41.325401 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0924 21:11:41.325404 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 21:11:41.325575 (+ 171us) server_negotiation.cc:407] Connection header received
0924 21:11:41.326409 (+ 834us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:11:41.326410 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:11:41.326419 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:11:41.326472 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:11:41.326962 (+ 490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:11:41.327419 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:11:41.328261 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:11:41.328520 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:11:41.328589 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:11:41.330417 (+ 1828us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 21:11:41.330418 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 21:11:41.330418 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 21:11:41.330423 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 21:11:41.331464 (+ 1041us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:11:41.331930 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:11:41.331931 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:11:41.331931 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:11:41.331993 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:11:41.332248 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:11:41.332248 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:11:41.332249 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:11:41.332373 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 21:11:41.332441 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0924 21:11:41.332746 (+ 305us) server_negotiation.cc:299] Negotiation successful
0924 21:11:41.332824 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260924 21:11:41.333163 19702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:11:41.325342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44011 (local address 127.0.98.1:38221)
0924 21:11:41.325466 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0924 21:11:41.325493 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0924 21:11:41.326325 (+ 832us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 21:11:41.326518 (+ 193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 21:11:41.326521 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 21:11:41.326739 (+ 218us) client_negotiation.cc:189] Negotiated authn=SASL
0924 21:11:41.326893 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:11:41.326897 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:11:41.327485 (+ 588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:11:41.327486 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:11:41.328191 (+ 705us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:11:41.328193 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:11:41.328602 (+ 409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:11:41.328603 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:11:41.328689 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:11:41.329059 (+ 370us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 21:11:41.329066 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0924 21:11:41.330347 (+ 1281us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 21:11:41.331538 (+ 1191us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 21:11:41.331545 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 21:11:41.331548 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 21:11:41.331853 (+ 305us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 21:11:41.332069 (+ 216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 21:11:41.332070 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 21:11:41.332070 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 21:11:41.332184 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 21:11:41.332443 (+ 259us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 21:11:41.332449 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 21:11:41.332610 (+ 161us) client_negotiation.cc:769] Sending connection context
0924 21:11:41.332718 (+ 108us) client_negotiation.cc:240] Negotiation successful
0924 21:11:41.332902 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":94,"thread_start_us":74,"threads_started":1}
I20260924 21:11:41.333729 19700 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44011
I20260924 21:11:41.333851 19700 heartbeater.cc:464] Registering TS with master...
I20260924 21:11:41.334023 19700 heartbeater.cc:511] Master 127.0.98.62:44011 requested a full tablet report, sending...
I20260924 21:11:41.334455 19509 ts_manager.cc:194] Registered new tserver with Master: d23fa4b8d26a4a87bcb9817236b7f1a2 (127.0.98.1:45995)
I20260924 21:11:41.334944 19509 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:38221
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790284076800039-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790284076800039-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260924 21:11:41.354147 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAzNNjS/build/release/bin/kudu
/tmp/dist-test-taskAzNNjS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790284076800039-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790284076800039-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790284076800039-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790284076800039-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44011
--keytab_file=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790284076800039-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34603
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790284076800039-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790284076800039-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790284076800039-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 21:11:41.423168 19705 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 21:11:41.423322 19705 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 21:11:41.423341 19705 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 21:11:41.424288 19705 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 21:11:41.424324 19705 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 21:11:41.424346 19705 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 21:11:41.425582 19705 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34603
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790284076800039-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790284076800039-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790284076800039-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790284076800039-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44011
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19705
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790284076800039-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 19:49:06 UTC on ccycloud.quasar-vhqjoc.root.comops.site
build id 4670864
I20260924 21:11:41.425851 19705 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 21:11:41.426069 19705 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 21:11:41.430501 19711 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 21:11:41.430495 19710 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 21:11:41.430495 19713 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 21:11:41.430665 19705 server_base.cc:1034] running on GCE node
I20260924 21:11:41.430886 19705 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 21:11:41.431103 19705 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 21:11:41.432227 19705 hybrid_clock.cc:648] HybridClock initialized: now 1790284301432212 us; error 31 us; skew 500 ppm
Sep 24 21:11:41 dist-test-slave-c7tk krb5kdc[19464](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284301, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 21:11:41.434366 19705 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 21:11:41.434762 19705 webserver.cc:460] Webserver started at http://127.0.98.2:38635/ using document root <none> and password file <none>
I20260924 21:11:41.434952 19705 fs_manager.cc:359] Metadata directory not provided
I20260924 21:11:41.434988 19705 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 21:11:41.435115 19705 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 21:11:41.435901 19705 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790284076800039-392-0/minicluster-data/ts-1/data/instance:
uuid: "10e88b55c0c34968b9e1ea6fe49ac885"
format_stamp: "Formatted at 2026-09-24 21:11:41 on dist-test-slave-c7tk"
server_key: "e24911ff995e6dfb3859a03bf8cfb5a1"
server_key_iv: "0961687cab0265e06f3b4cb387c3d09a"
server_key_version: "encryptionkey@0"
I20260924 21:11:41.436189 19705 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790284076800039-392-0/minicluster-data/ts-1/wal/instance:
uuid: "10e88b55c0c34968b9e1ea6fe49ac885"
format_stamp: "Formatted at 2026-09-24 21:11:41 on dist-test-slave-c7tk"
server_key: "e24911ff995e6dfb3859a03bf8cfb5a1"
server_key_iv: "0961687cab0265e06f3b4cb387c3d09a"
server_key_version: "encryptionkey@0"
I20260924 21:11:41.437392 19705 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260924 21:11:41.438002 19720 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 21:11:41.438139 19705 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 21:11:41.438197 19705 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790284076800039-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790284076800039-392-0/minicluster-data/ts-1/wal
uuid: "10e88b55c0c34968b9e1ea6fe49ac885"
format_stamp: "Formatted at 2026-09-24 21:11:41 on dist-test-slave-c7tk"
server_key: "e24911ff995e6dfb3859a03bf8cfb5a1"
server_key_iv: "0961687cab0265e06f3b4cb387c3d09a"
server_key_version: "encryptionkey@0"
I20260924 21:11:41.438282 19705 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790284076800039-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790284076800039-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790284076800039-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 21:11:41.445555 19705 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 21:11:41.447120 19705 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 21:11:41.447252 19705 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 21:11:41.447420 19705 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 21:11:41.447659 19705 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 21:11:41.447687 19705 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 21:11:41.447705 19705 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 21:11:41.447732 19705 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 21:11:41.453262 19705 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:41403
I20260924 21:11:41.453315 19833 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:41403 every 8 connection(s)
I20260924 21:11:41.453577 19705 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790284076800039-392-0/minicluster-data/ts-1/data/info.pb
I20260924 21:11:41.457962 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAzNNjS/build/release/bin/kudu as pid 19705
I20260924 21:11:41.458067 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790284076800039-392-0/minicluster-data/ts-1/wal/instance
I20260924 21:11:41.458221 392 external_mini_cluster.cc:1428] Setting key c8633bd5b37447d112738a11d2e59f8b
Sep 24 21:11:41 dist-test-slave-c7tk krb5kdc[19464](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284301, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 21:11:41.461902 19548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:11:41.454227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:52003 (local address 127.0.98.62:44011)
0924 21:11:41.454295 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0924 21:11:41.454297 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 21:11:41.454421 (+ 124us) server_negotiation.cc:407] Connection header received
0924 21:11:41.455188 (+ 767us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:11:41.455188 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:11:41.455197 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:11:41.455241 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:11:41.455698 (+ 457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:11:41.456167 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:11:41.457000 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:11:41.457153 (+ 153us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:11:41.457195 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:11:41.459177 (+ 1982us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 21:11:41.459178 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 21:11:41.459179 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 21:11:41.459186 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 21:11:41.460611 (+ 1425us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:11:41.461005 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:11:41.461006 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:11:41.461006 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:11:41.461047 (+ 41us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:11:41.461285 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:11:41.461285 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:11:41.461285 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:11:41.461365 (+ 80us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 21:11:41.461405 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0924 21:11:41.461704 (+ 299us) server_negotiation.cc:299] Negotiation successful
0924 21:11:41.461780 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260924 21:11:41.462070 19836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:11:41.454184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44011 (local address 127.0.98.2:52003)
0924 21:11:41.454316 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0924 21:11:41.454343 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0924 21:11:41.455110 (+ 767us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 21:11:41.455282 (+ 172us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 21:11:41.455285 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 21:11:41.455480 (+ 195us) client_negotiation.cc:189] Negotiated authn=SASL
0924 21:11:41.455636 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:11:41.455639 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:11:41.456244 (+ 605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:11:41.456245 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:11:41.456938 (+ 693us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:11:41.456940 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:11:41.457211 (+ 271us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:11:41.457212 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:11:41.457290 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:11:41.457622 (+ 332us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 21:11:41.457628 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0924 21:11:41.459091 (+ 1463us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 21:11:41.460674 (+ 1583us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 21:11:41.460679 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 21:11:41.460681 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 21:11:41.460936 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 21:11:41.461106 (+ 170us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 21:11:41.461106 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 21:11:41.461107 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 21:11:41.461217 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 21:11:41.461419 (+ 202us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 21:11:41.461440 (+ 21us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 21:11:41.461573 (+ 133us) client_negotiation.cc:769] Sending connection context
0924 21:11:41.461666 (+ 93us) client_negotiation.cc:240] Negotiation successful
0924 21:11:41.461792 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":78,"threads_started":1}
I20260924 21:11:41.462579 19834 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44011
I20260924 21:11:41.462641 19834 heartbeater.cc:464] Registering TS with master...
I20260924 21:11:41.462787 19834 heartbeater.cc:511] Master 127.0.98.62:44011 requested a full tablet report, sending...
I20260924 21:11:41.463135 19509 ts_manager.cc:194] Registered new tserver with Master: 10e88b55c0c34968b9e1ea6fe49ac885 (127.0.98.2:41403)
I20260924 21:11:41.463610 19509 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:52003
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790284076800039-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790284076800039-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260924 21:11:41.480859 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAzNNjS/build/release/bin/kudu
/tmp/dist-test-taskAzNNjS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790284076800039-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790284076800039-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790284076800039-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790284076800039-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44011
--keytab_file=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790284076800039-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34603
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790284076800039-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790284076800039-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790284076800039-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 21:11:41.553169 19839 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 21:11:41.553329 19839 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 21:11:41.553349 19839 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 21:11:41.554426 19839 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 21:11:41.554463 19839 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 21:11:41.554492 19839 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 21:11:41.555789 19839 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34603
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790284076800039-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790284076800039-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790284076800039-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790284076800039-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44011
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19839
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790284076800039-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 19:49:06 UTC on ccycloud.quasar-vhqjoc.root.comops.site
build id 4670864
I20260924 21:11:41.555991 19839 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 21:11:41.556185 19839 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 21:11:41.560830 19844 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 21:11:41.560864 19847 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 21:11:41.560912 19839 server_base.cc:1034] running on GCE node
W20260924 21:11:41.560882 19845 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 21:11:41.561199 19839 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 21:11:41.561367 19839 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 21:11:41.562492 19839 hybrid_clock.cc:648] HybridClock initialized: now 1790284301562477 us; error 31 us; skew 500 ppm
Sep 24 21:11:41 dist-test-slave-c7tk krb5kdc[19464](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284301, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 21:11:41.564582 19839 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 21:11:41.564925 19839 webserver.cc:460] Webserver started at http://127.0.98.3:40809/ using document root <none> and password file <none>
I20260924 21:11:41.565115 19839 fs_manager.cc:359] Metadata directory not provided
I20260924 21:11:41.565153 19839 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 21:11:41.565258 19839 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 21:11:41.566128 19839 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790284076800039-392-0/minicluster-data/ts-2/data/instance:
uuid: "782e9d502c824b62af4770b0b466cc2a"
format_stamp: "Formatted at 2026-09-24 21:11:41 on dist-test-slave-c7tk"
server_key: "6c2a4606def6db7544a1bcbbb5e15722"
server_key_iv: "4f3b7c7501cdbc990f170ff0dbb1bb5a"
server_key_version: "encryptionkey@0"
I20260924 21:11:41.566416 19839 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790284076800039-392-0/minicluster-data/ts-2/wal/instance:
uuid: "782e9d502c824b62af4770b0b466cc2a"
format_stamp: "Formatted at 2026-09-24 21:11:41 on dist-test-slave-c7tk"
server_key: "6c2a4606def6db7544a1bcbbb5e15722"
server_key_iv: "4f3b7c7501cdbc990f170ff0dbb1bb5a"
server_key_version: "encryptionkey@0"
I20260924 21:11:41.567466 19839 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 21:11:41.568040 19854 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 21:11:41.568176 19839 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 21:11:41.568259 19839 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790284076800039-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790284076800039-392-0/minicluster-data/ts-2/wal
uuid: "782e9d502c824b62af4770b0b466cc2a"
format_stamp: "Formatted at 2026-09-24 21:11:41 on dist-test-slave-c7tk"
server_key: "6c2a4606def6db7544a1bcbbb5e15722"
server_key_iv: "4f3b7c7501cdbc990f170ff0dbb1bb5a"
server_key_version: "encryptionkey@0"
I20260924 21:11:41.568336 19839 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790284076800039-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790284076800039-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790284076800039-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 21:11:41.579267 19839 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 21:11:41.580849 19839 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 21:11:41.580982 19839 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 21:11:41.581161 19839 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 21:11:41.581398 19839 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 21:11:41.581444 19839 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 21:11:41.581476 19839 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 21:11:41.581498 19839 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 21:11:41.586774 19839 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:44861
I20260924 21:11:41.586809 19967 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:44861 every 8 connection(s)
I20260924 21:11:41.587095 19839 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790284076800039-392-0/minicluster-data/ts-2/data/info.pb
Sep 24 21:11:41 dist-test-slave-c7tk krb5kdc[19464](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284301, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 21:11:41.594416 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAzNNjS/build/release/bin/kudu as pid 19839
I20260924 21:11:41.594492 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790284076800039-392-0/minicluster-data/ts-2/wal/instance
I20260924 21:11:41.594673 392 external_mini_cluster.cc:1428] Setting key 46006c2cf4dcf15f6e8b96919fcb7d08
I20260924 21:11:41.595603 19548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:11:41.587723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:35453 (local address 127.0.98.62:44011)
0924 21:11:41.587764 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0924 21:11:41.587766 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 21:11:41.587918 (+ 152us) server_negotiation.cc:407] Connection header received
0924 21:11:41.588663 (+ 745us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:11:41.588664 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:11:41.588673 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:11:41.588725 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:11:41.589245 (+ 520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:11:41.589773 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:11:41.590764 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:11:41.590947 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:11:41.590996 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:11:41.592929 (+ 1933us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 21:11:41.592931 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 21:11:41.592931 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 21:11:41.592937 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 21:11:41.594217 (+ 1280us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:11:41.594616 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:11:41.594618 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:11:41.594618 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:11:41.594675 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:11:41.594991 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:11:41.594992 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:11:41.594992 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:11:41.595108 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 21:11:41.595179 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0924 21:11:41.595433 (+ 254us) server_negotiation.cc:299] Negotiation successful
0924 21:11:41.595508 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260924 21:11:41.595688 19970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:11:41.587681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44011 (local address 127.0.98.3:35453)
0924 21:11:41.587807 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0924 21:11:41.587840 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0924 21:11:41.588579 (+ 739us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 21:11:41.588769 (+ 190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 21:11:41.588772 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 21:11:41.588989 (+ 217us) client_negotiation.cc:189] Negotiated authn=SASL
0924 21:11:41.589176 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:11:41.589179 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:11:41.589926 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:11:41.589927 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:11:41.590691 (+ 764us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:11:41.590693 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:11:41.591015 (+ 322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:11:41.591016 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:11:41.591128 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:11:41.591487 (+ 359us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 21:11:41.591496 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 21:11:41.592854 (+ 1358us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 21:11:41.594319 (+ 1465us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 21:11:41.594323 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 21:11:41.594324 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 21:11:41.594550 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 21:11:41.594742 (+ 192us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 21:11:41.594743 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 21:11:41.594743 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 21:11:41.594873 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 21:11:41.595172 (+ 299us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 21:11:41.595177 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 21:11:41.595311 (+ 134us) client_negotiation.cc:769] Sending connection context
0924 21:11:41.595399 (+ 88us) client_negotiation.cc:240] Negotiation successful
0924 21:11:41.595520 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":73,"threads_started":1}
I20260924 21:11:41.596068 19968 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44011
I20260924 21:11:41.596161 19968 heartbeater.cc:464] Registering TS with master...
I20260924 21:11:41.596318 19968 heartbeater.cc:511] Master 127.0.98.62:44011 requested a full tablet report, sending...
I20260924 21:11:41.596736 19509 ts_manager.cc:194] Registered new tserver with Master: 782e9d502c824b62af4770b0b466cc2a (127.0.98.3:44861)
I20260924 21:11:41.597149 19509 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:35453
I20260924 21:11:41.606304 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 21:11:41.612402 19548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:11:41.607494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40372 (local address 127.0.98.62:44011)
0924 21:11:41.607556 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0924 21:11:41.607557 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 21:11:41.607634 (+ 77us) server_negotiation.cc:407] Connection header received
0924 21:11:41.607662 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:11:41.607663 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:11:41.607670 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:11:41.607713 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:11:41.608252 (+ 539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:11:41.608751 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:11:41.609398 (+ 647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:11:41.609573 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:11:41.609624 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:11:41.610675 (+ 1051us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 21:11:41.610675 (+ 0us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 21:11:41.610676 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 21:11:41.610680 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 21:11:41.611649 (+ 969us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:11:41.611894 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:11:41.611894 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:11:41.611895 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:11:41.611933 (+ 38us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:11:41.612051 (+ 118us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:11:41.612051 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:11:41.612051 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:11:41.612147 (+ 96us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 21:11:41.612206 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0924 21:11:41.612276 (+ 70us) server_negotiation.cc:299] Negotiation successful
0924 21:11:41.612323 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260924 21:11:41.613777 19509 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40372:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 21:11:41.615142 19509 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 21:11:41.621235 19976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:11:41.616944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45995 (local address 127.0.0.1:35710)
0924 21:11:41.617220 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0924 21:11:41.617231 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 21:11:41.617283 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 21:11:41.617528 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 21:11:41.617531 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 21:11:41.617534 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 21:11:41.617598 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:11:41.617600 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:11:41.618850 (+ 1250us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:11:41.618851 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:11:41.620181 (+ 1330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:11:41.620184 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:11:41.620940 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:11:41.620941 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:11:41.621011 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:11:41.621030 (+ 19us) client_negotiation.cc:769] Sending connection context
0924 21:11:41.621101 (+ 71us) client_negotiation.cc:240] Negotiation successful
0924 21:11:41.621151 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"mutex_wait_us":66,"thread_start_us":76,"threads_started":1}
I20260924 21:11:41.621243 19978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:11:41.617068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35710 (local address 127.0.98.1:45995)
0924 21:11:41.617368 (+ 300us) server_negotiation.cc:206] Beginning negotiation
0924 21:11:41.617372 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 21:11:41.617382 (+ 10us) server_negotiation.cc:407] Connection header received
0924 21:11:41.617418 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:11:41.617420 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:11:41.617434 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:11:41.617505 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 21:11:41.617680 (+ 175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:11:41.618728 (+ 1048us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:11:41.620257 (+ 1529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:11:41.620861 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:11:41.620911 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:11:41.620945 (+ 34us) server_negotiation.cc:1035] Waiting for connection context
0924 21:11:41.621134 (+ 189us) server_negotiation.cc:299] Negotiation successful
0924 21:11:41.621172 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":262,"thread_start_us":62,"threads_started":1}
I20260924 21:11:41.621516 19977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:11:41.616939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44861 (local address 127.0.0.1:56242)
0924 21:11:41.617067 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0924 21:11:41.617099 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0924 21:11:41.617150 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 21:11:41.618872 (+ 1722us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 21:11:41.618873 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 21:11:41.618875 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 21:11:41.618917 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:11:41.618918 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:11:41.619764 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:11:41.619765 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:11:41.620581 (+ 816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:11:41.620583 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:11:41.621297 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:11:41.621298 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:11:41.621371 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:11:41.621374 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 21:11:41.621422 (+ 48us) client_negotiation.cc:240] Negotiation successful
0924 21:11:41.621459 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101,"mutex_wait_us":1,"thread_start_us":60,"threads_started":1}
I20260924 21:11:41.621538 19979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:11:41.617106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56242 (local address 127.0.98.3:44861)
0924 21:11:41.617626 (+ 520us) server_negotiation.cc:206] Beginning negotiation
0924 21:11:41.617630 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 21:11:41.617638 (+ 8us) server_negotiation.cc:407] Connection header received
0924 21:11:41.617664 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:11:41.617666 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:11:41.617679 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:11:41.617755 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 21:11:41.618991 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:11:41.619635 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:11:41.620656 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:11:41.621165 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:11:41.621220 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:11:41.621266 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0924 21:11:41.621437 (+ 171us) server_negotiation.cc:299] Negotiation successful
0924 21:11:41.621472 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":478,"thread_start_us":64,"threads_started":1}
I20260924 21:11:41.622082 19980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:11:41.617139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41403 (local address 127.0.0.1:54574)
0924 21:11:41.617242 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0924 21:11:41.617252 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 21:11:41.617287 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 21:11:41.617647 (+ 360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 21:11:41.617649 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 21:11:41.617650 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 21:11:41.617691 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:11:41.617692 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:11:41.618867 (+ 1175us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:11:41.618867 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:11:41.620181 (+ 1314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:11:41.620184 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:11:41.621865 (+ 1681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:11:41.621866 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:11:41.621921 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:11:41.621925 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 21:11:41.621976 (+ 51us) client_negotiation.cc:240] Negotiation successful
0924 21:11:41.622011 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":89,"mutex_wait_us":40,"thread_start_us":70,"threads_started":1}
I20260924 21:11:41.622140 19634 tablet_service.cc:1467] Processing CreateTablet for tablet a58adae0b7574cef8c19e5b2b0781603 (DEFAULT_TABLE table=test-table [id=74389c9843a14932897d443e8b7dc395]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 21:11:41.622200 19902 tablet_service.cc:1467] Processing CreateTablet for tablet a58adae0b7574cef8c19e5b2b0781603 (DEFAULT_TABLE table=test-table [id=74389c9843a14932897d443e8b7dc395]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 21:11:41.622359 19981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:11:41.617317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54574 (local address 127.0.98.2:41403)
0924 21:11:41.617477 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0924 21:11:41.617497 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0924 21:11:41.617506 (+ 9us) server_negotiation.cc:407] Connection header received
0924 21:11:41.617538 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:11:41.617540 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:11:41.617554 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:11:41.617631 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 21:11:41.617790 (+ 159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:11:41.618764 (+ 974us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:11:41.620972 (+ 2208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:11:41.621625 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:11:41.621720 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:11:41.621781 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0924 21:11:41.622234 (+ 453us) server_negotiation.cc:299] Negotiation successful
0924 21:11:41.622282 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":86,"threads_started":1}
I20260924 21:11:41.622486 19634 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a58adae0b7574cef8c19e5b2b0781603. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 21:11:41.622506 19902 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a58adae0b7574cef8c19e5b2b0781603. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 21:11:41.624531 19982 tablet_bootstrap.cc:492] T a58adae0b7574cef8c19e5b2b0781603 P 782e9d502c824b62af4770b0b466cc2a: Bootstrap starting.
I20260924 21:11:41.624881 19768 tablet_service.cc:1467] Processing CreateTablet for tablet a58adae0b7574cef8c19e5b2b0781603 (DEFAULT_TABLE table=test-table [id=74389c9843a14932897d443e8b7dc395]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 21:11:41.625154 19982 tablet_bootstrap.cc:654] T a58adae0b7574cef8c19e5b2b0781603 P 782e9d502c824b62af4770b0b466cc2a: Neither blocks nor log segments found. Creating new log.
I20260924 21:11:41.625159 19768 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a58adae0b7574cef8c19e5b2b0781603. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 21:11:41.625655 19982 log.cc:824] T a58adae0b7574cef8c19e5b2b0781603 P 782e9d502c824b62af4770b0b466cc2a: Log is configured to *not* fsync() on all Append() calls
I20260924 21:11:41.625943 19983 tablet_bootstrap.cc:492] T a58adae0b7574cef8c19e5b2b0781603 P d23fa4b8d26a4a87bcb9817236b7f1a2: Bootstrap starting.
I20260924 21:11:41.626318 19982 tablet_bootstrap.cc:492] T a58adae0b7574cef8c19e5b2b0781603 P 782e9d502c824b62af4770b0b466cc2a: No bootstrap required, opened a new log
I20260924 21:11:41.626382 19982 ts_tablet_manager.cc:1397] T a58adae0b7574cef8c19e5b2b0781603 P 782e9d502c824b62af4770b0b466cc2a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 21:11:41.626514 19983 tablet_bootstrap.cc:654] T a58adae0b7574cef8c19e5b2b0781603 P d23fa4b8d26a4a87bcb9817236b7f1a2: Neither blocks nor log segments found. Creating new log.
I20260924 21:11:41.626981 19983 log.cc:824] T a58adae0b7574cef8c19e5b2b0781603 P d23fa4b8d26a4a87bcb9817236b7f1a2: Log is configured to *not* fsync() on all Append() calls
I20260924 21:11:41.627703 19986 tablet_bootstrap.cc:492] T a58adae0b7574cef8c19e5b2b0781603 P 10e88b55c0c34968b9e1ea6fe49ac885: Bootstrap starting.
I20260924 21:11:41.627679 19983 tablet_bootstrap.cc:492] T a58adae0b7574cef8c19e5b2b0781603 P d23fa4b8d26a4a87bcb9817236b7f1a2: No bootstrap required, opened a new log
I20260924 21:11:41.627902 19983 ts_tablet_manager.cc:1397] T a58adae0b7574cef8c19e5b2b0781603 P d23fa4b8d26a4a87bcb9817236b7f1a2: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 21:11:41.628304 19986 tablet_bootstrap.cc:654] T a58adae0b7574cef8c19e5b2b0781603 P 10e88b55c0c34968b9e1ea6fe49ac885: Neither blocks nor log segments found. Creating new log.
I20260924 21:11:41.628232 19982 raft_consensus.cc:348] T a58adae0b7574cef8c19e5b2b0781603 P 782e9d502c824b62af4770b0b466cc2a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "782e9d502c824b62af4770b0b466cc2a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44861 } } peers { permanent_uuid: "d23fa4b8d26a4a87bcb9817236b7f1a2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45995 } } peers { permanent_uuid: "10e88b55c0c34968b9e1ea6fe49ac885" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41403 } }
I20260924 21:11:41.628367 19982 raft_consensus.cc:374] T a58adae0b7574cef8c19e5b2b0781603 P 782e9d502c824b62af4770b0b466cc2a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 21:11:41.628407 19982 raft_consensus.cc:729] T a58adae0b7574cef8c19e5b2b0781603 P 782e9d502c824b62af4770b0b466cc2a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 782e9d502c824b62af4770b0b466cc2a, State: Initialized, Role: FOLLOWER
I20260924 21:11:41.628587 19982 consensus_queue.cc:260] T a58adae0b7574cef8c19e5b2b0781603 P 782e9d502c824b62af4770b0b466cc2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "782e9d502c824b62af4770b0b466cc2a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44861 } } peers { permanent_uuid: "d23fa4b8d26a4a87bcb9817236b7f1a2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45995 } } peers { permanent_uuid: "10e88b55c0c34968b9e1ea6fe49ac885" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41403 } }
I20260924 21:11:41.628845 19982 ts_tablet_manager.cc:1428] T a58adae0b7574cef8c19e5b2b0781603 P 782e9d502c824b62af4770b0b466cc2a: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260924 21:11:41.628885 19968 heartbeater.cc:503] Master 127.0.98.62:44011 was elected leader, sending a full tablet report...
I20260924 21:11:41.628902 19986 log.cc:824] T a58adae0b7574cef8c19e5b2b0781603 P 10e88b55c0c34968b9e1ea6fe49ac885: Log is configured to *not* fsync() on all Append() calls
I20260924 21:11:41.629738 19986 tablet_bootstrap.cc:492] T a58adae0b7574cef8c19e5b2b0781603 P 10e88b55c0c34968b9e1ea6fe49ac885: No bootstrap required, opened a new log
I20260924 21:11:41.629815 19986 ts_tablet_manager.cc:1397] T a58adae0b7574cef8c19e5b2b0781603 P 10e88b55c0c34968b9e1ea6fe49ac885: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 21:11:41.629896 19983 raft_consensus.cc:348] T a58adae0b7574cef8c19e5b2b0781603 P d23fa4b8d26a4a87bcb9817236b7f1a2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "782e9d502c824b62af4770b0b466cc2a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44861 } } peers { permanent_uuid: "d23fa4b8d26a4a87bcb9817236b7f1a2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45995 } } peers { permanent_uuid: "10e88b55c0c34968b9e1ea6fe49ac885" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41403 } }
I20260924 21:11:41.630077 19983 raft_consensus.cc:374] T a58adae0b7574cef8c19e5b2b0781603 P d23fa4b8d26a4a87bcb9817236b7f1a2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 21:11:41.630121 19983 raft_consensus.cc:729] T a58adae0b7574cef8c19e5b2b0781603 P d23fa4b8d26a4a87bcb9817236b7f1a2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d23fa4b8d26a4a87bcb9817236b7f1a2, State: Initialized, Role: FOLLOWER
I20260924 21:11:41.630216 19983 consensus_queue.cc:260] T a58adae0b7574cef8c19e5b2b0781603 P d23fa4b8d26a4a87bcb9817236b7f1a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "782e9d502c824b62af4770b0b466cc2a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44861 } } peers { permanent_uuid: "d23fa4b8d26a4a87bcb9817236b7f1a2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45995 } } peers { permanent_uuid: "10e88b55c0c34968b9e1ea6fe49ac885" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41403 } }
I20260924 21:11:41.630430 19983 ts_tablet_manager.cc:1428] T a58adae0b7574cef8c19e5b2b0781603 P d23fa4b8d26a4a87bcb9817236b7f1a2: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 21:11:41.630440 19700 heartbeater.cc:503] Master 127.0.98.62:44011 was elected leader, sending a full tablet report...
I20260924 21:11:41.631477 19986 raft_consensus.cc:348] T a58adae0b7574cef8c19e5b2b0781603 P 10e88b55c0c34968b9e1ea6fe49ac885 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "782e9d502c824b62af4770b0b466cc2a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44861 } } peers { permanent_uuid: "d23fa4b8d26a4a87bcb9817236b7f1a2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45995 } } peers { permanent_uuid: "10e88b55c0c34968b9e1ea6fe49ac885" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41403 } }
I20260924 21:11:41.631611 19986 raft_consensus.cc:374] T a58adae0b7574cef8c19e5b2b0781603 P 10e88b55c0c34968b9e1ea6fe49ac885 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 21:11:41.631650 19986 raft_consensus.cc:729] T a58adae0b7574cef8c19e5b2b0781603 P 10e88b55c0c34968b9e1ea6fe49ac885 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10e88b55c0c34968b9e1ea6fe49ac885, State: Initialized, Role: FOLLOWER
I20260924 21:11:41.631739 19986 consensus_queue.cc:260] T a58adae0b7574cef8c19e5b2b0781603 P 10e88b55c0c34968b9e1ea6fe49ac885 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "782e9d502c824b62af4770b0b466cc2a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44861 } } peers { permanent_uuid: "d23fa4b8d26a4a87bcb9817236b7f1a2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45995 } } peers { permanent_uuid: "10e88b55c0c34968b9e1ea6fe49ac885" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41403 } }
I20260924 21:11:41.631963 19986 ts_tablet_manager.cc:1428] T a58adae0b7574cef8c19e5b2b0781603 P 10e88b55c0c34968b9e1ea6fe49ac885: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 21:11:41.631992 19834 heartbeater.cc:503] Master 127.0.98.62:44011 was elected leader, sending a full tablet report...
W20260924 21:11:41.704087 19835 tablet.cc:2369] T a58adae0b7574cef8c19e5b2b0781603 P 10e88b55c0c34968b9e1ea6fe49ac885: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 21:11:41.792758 19989 raft_consensus.cc:482] T a58adae0b7574cef8c19e5b2b0781603 P d23fa4b8d26a4a87bcb9817236b7f1a2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 21:11:41.792891 19989 raft_consensus.cc:504] T a58adae0b7574cef8c19e5b2b0781603 P d23fa4b8d26a4a87bcb9817236b7f1a2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "782e9d502c824b62af4770b0b466cc2a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44861 } } peers { permanent_uuid: "d23fa4b8d26a4a87bcb9817236b7f1a2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45995 } } peers { permanent_uuid: "10e88b55c0c34968b9e1ea6fe49ac885" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41403 } }
I20260924 21:11:41.793273 19989 leader_election.cc:288] T a58adae0b7574cef8c19e5b2b0781603 P d23fa4b8d26a4a87bcb9817236b7f1a2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 782e9d502c824b62af4770b0b466cc2a (127.0.98.3:44861), 10e88b55c0c34968b9e1ea6fe49ac885 (127.0.98.2:41403)
I20260924 21:11:41.796016 19979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:11:41.793528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:51485 (local address 127.0.98.3:44861)
0924 21:11:41.793576 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0924 21:11:41.793578 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 21:11:41.793589 (+ 11us) server_negotiation.cc:407] Connection header received
0924 21:11:41.793668 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:11:41.793669 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:11:41.793678 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:11:41.793734 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 21:11:41.793932 (+ 198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:11:41.794437 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:11:41.795316 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:11:41.795668 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:11:41.795717 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:11:41.795725 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0924 21:11:41.795883 (+ 158us) server_negotiation.cc:299] Negotiation successful
0924 21:11:41.795944 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260924 21:11:41.796029 19702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:11:41.793436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44861 (local address 127.0.98.1:51485)
0924 21:11:41.793507 (+ 71us) negotiation.cc:107] Waiting for socket to connect
0924 21:11:41.793521 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 21:11:41.793582 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 21:11:41.793741 (+ 159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 21:11:41.793742 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 21:11:41.793744 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 21:11:41.793833 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:11:41.793835 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:11:41.794505 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:11:41.794506 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:11:41.795229 (+ 723us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:11:41.795231 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:11:41.795739 (+ 508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:11:41.795740 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:11:41.795814 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:11:41.795817 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 21:11:41.795870 (+ 53us) client_negotiation.cc:240] Negotiation successful
0924 21:11:41.795917 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45}
I20260924 21:11:41.796084 19981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:11:41.793652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:50483 (local address 127.0.98.2:41403)
0924 21:11:41.793696 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0924 21:11:41.793698 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 21:11:41.793789 (+ 91us) server_negotiation.cc:407] Connection header received
0924 21:11:41.793858 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:11:41.793859 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:11:41.793869 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:11:41.793926 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 21:11:41.794099 (+ 173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:11:41.794513 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:11:41.795369 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:11:41.795689 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:11:41.795769 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:11:41.795778 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0924 21:11:41.795968 (+ 190us) server_negotiation.cc:299] Negotiation successful
0924 21:11:41.796011 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260924 21:11:41.796335 19991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:11:41.793568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41403 (local address 127.0.98.1:50483)
0924 21:11:41.793732 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0924 21:11:41.793741 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 21:11:41.793787 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 21:11:41.793948 (+ 161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 21:11:41.793949 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 21:11:41.793951 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 21:11:41.794037 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:11:41.794038 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:11:41.794591 (+ 553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:11:41.794592 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:11:41.795281 (+ 689us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:11:41.795283 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:11:41.795797 (+ 514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:11:41.795798 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:11:41.795870 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:11:41.795874 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 21:11:41.795946 (+ 72us) client_negotiation.cc:240] Negotiation successful
0924 21:11:41.795982 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"mutex_wait_us":8,"thread_start_us":86,"threads_started":1}
I20260924 21:11:41.796397 19922 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a58adae0b7574cef8c19e5b2b0781603" candidate_uuid: "d23fa4b8d26a4a87bcb9817236b7f1a2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "782e9d502c824b62af4770b0b466cc2a" is_pre_election: true
I20260924 21:11:41.796545 19922 raft_consensus.cc:2454] T a58adae0b7574cef8c19e5b2b0781603 P 782e9d502c824b62af4770b0b466cc2a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d23fa4b8d26a4a87bcb9817236b7f1a2 in term 0.
I20260924 21:11:41.796672 19788 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a58adae0b7574cef8c19e5b2b0781603" candidate_uuid: "d23fa4b8d26a4a87bcb9817236b7f1a2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10e88b55c0c34968b9e1ea6fe49ac885" is_pre_election: true
I20260924 21:11:41.796774 19589 leader_election.cc:302] T a58adae0b7574cef8c19e5b2b0781603 P d23fa4b8d26a4a87bcb9817236b7f1a2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 782e9d502c824b62af4770b0b466cc2a, d23fa4b8d26a4a87bcb9817236b7f1a2; no voters:
I20260924 21:11:41.796823 19788 raft_consensus.cc:2454] T a58adae0b7574cef8c19e5b2b0781603 P 10e88b55c0c34968b9e1ea6fe49ac885 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d23fa4b8d26a4a87bcb9817236b7f1a2 in term 0.
I20260924 21:11:41.796885 19989 raft_consensus.cc:2781] T a58adae0b7574cef8c19e5b2b0781603 P d23fa4b8d26a4a87bcb9817236b7f1a2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 21:11:41.796950 19989 raft_consensus.cc:482] T a58adae0b7574cef8c19e5b2b0781603 P d23fa4b8d26a4a87bcb9817236b7f1a2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 21:11:41.796980 19989 raft_consensus.cc:3037] T a58adae0b7574cef8c19e5b2b0781603 P d23fa4b8d26a4a87bcb9817236b7f1a2 [term 0 FOLLOWER]: Advancing to term 1
I20260924 21:11:41.797658 19989 raft_consensus.cc:504] T a58adae0b7574cef8c19e5b2b0781603 P d23fa4b8d26a4a87bcb9817236b7f1a2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "782e9d502c824b62af4770b0b466cc2a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44861 } } peers { permanent_uuid: "d23fa4b8d26a4a87bcb9817236b7f1a2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45995 } } peers { permanent_uuid: "10e88b55c0c34968b9e1ea6fe49ac885" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41403 } }
I20260924 21:11:41.797763 19989 leader_election.cc:288] T a58adae0b7574cef8c19e5b2b0781603 P d23fa4b8d26a4a87bcb9817236b7f1a2 [CANDIDATE]: Term 1 election: Requested vote from peers 782e9d502c824b62af4770b0b466cc2a (127.0.98.3:44861), 10e88b55c0c34968b9e1ea6fe49ac885 (127.0.98.2:41403)
I20260924 21:11:41.797960 19788 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a58adae0b7574cef8c19e5b2b0781603" candidate_uuid: "d23fa4b8d26a4a87bcb9817236b7f1a2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10e88b55c0c34968b9e1ea6fe49ac885"
I20260924 21:11:41.797961 19922 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a58adae0b7574cef8c19e5b2b0781603" candidate_uuid: "d23fa4b8d26a4a87bcb9817236b7f1a2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "782e9d502c824b62af4770b0b466cc2a"
I20260924 21:11:41.798022 19788 raft_consensus.cc:3037] T a58adae0b7574cef8c19e5b2b0781603 P 10e88b55c0c34968b9e1ea6fe49ac885 [term 0 FOLLOWER]: Advancing to term 1
I20260924 21:11:41.798054 19922 raft_consensus.cc:3037] T a58adae0b7574cef8c19e5b2b0781603 P 782e9d502c824b62af4770b0b466cc2a [term 0 FOLLOWER]: Advancing to term 1
I20260924 21:11:41.798753 19922 raft_consensus.cc:2454] T a58adae0b7574cef8c19e5b2b0781603 P 782e9d502c824b62af4770b0b466cc2a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d23fa4b8d26a4a87bcb9817236b7f1a2 in term 1.
I20260924 21:11:41.798905 19788 raft_consensus.cc:2454] T a58adae0b7574cef8c19e5b2b0781603 P 10e88b55c0c34968b9e1ea6fe49ac885 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d23fa4b8d26a4a87bcb9817236b7f1a2 in term 1.
I20260924 21:11:41.798913 19589 leader_election.cc:302] T a58adae0b7574cef8c19e5b2b0781603 P d23fa4b8d26a4a87bcb9817236b7f1a2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 782e9d502c824b62af4770b0b466cc2a, d23fa4b8d26a4a87bcb9817236b7f1a2; no voters:
I20260924 21:11:41.799118 19989 raft_consensus.cc:2781] T a58adae0b7574cef8c19e5b2b0781603 P d23fa4b8d26a4a87bcb9817236b7f1a2 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 21:11:41.799273 19989 raft_consensus.cc:686] T a58adae0b7574cef8c19e5b2b0781603 P d23fa4b8d26a4a87bcb9817236b7f1a2 [term 1 LEADER]: Becoming Leader. State: Replica: d23fa4b8d26a4a87bcb9817236b7f1a2, State: Running, Role: LEADER
I20260924 21:11:41.799371 19989 consensus_queue.cc:237] T a58adae0b7574cef8c19e5b2b0781603 P d23fa4b8d26a4a87bcb9817236b7f1a2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "782e9d502c824b62af4770b0b466cc2a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44861 } } peers { permanent_uuid: "d23fa4b8d26a4a87bcb9817236b7f1a2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45995 } } peers { permanent_uuid: "10e88b55c0c34968b9e1ea6fe49ac885" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41403 } }
I20260924 21:11:41.800151 19508 catalog_manager.cc:5495] T a58adae0b7574cef8c19e5b2b0781603 P d23fa4b8d26a4a87bcb9817236b7f1a2 reported cstate change: term changed from 0 to 1, leader changed from <none> to d23fa4b8d26a4a87bcb9817236b7f1a2 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "d23fa4b8d26a4a87bcb9817236b7f1a2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "782e9d502c824b62af4770b0b466cc2a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44861 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d23fa4b8d26a4a87bcb9817236b7f1a2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45995 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "10e88b55c0c34968b9e1ea6fe49ac885" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41403 } health_report { overall_health: UNKNOWN } } }
I20260924 21:11:41.804575 19978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:11:41.802851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35724 (local address 127.0.98.1:45995)
0924 21:11:41.802885 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0924 21:11:41.802887 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 21:11:41.802900 (+ 13us) server_negotiation.cc:407] Connection header received
0924 21:11:41.802948 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:11:41.802948 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:11:41.802961 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:11:41.803017 (+ 56us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 21:11:41.803109 (+ 92us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:11:41.803474 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:11:41.804064 (+ 590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:11:41.804220 (+ 156us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:11:41.804263 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:11:41.804364 (+ 101us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 21:11:41.804439 (+ 75us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 21:11:41.804486 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0924 21:11:41.804511 (+ 25us) server_negotiation.cc:299] Negotiation successful
0924 21:11:41.804523 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260924 21:11:41.806064 19788 raft_consensus.cc:1261] T a58adae0b7574cef8c19e5b2b0781603 P 10e88b55c0c34968b9e1ea6fe49ac885 [term 1 FOLLOWER]: Refusing update from remote peer d23fa4b8d26a4a87bcb9817236b7f1a2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 21:11:41.806068 19922 raft_consensus.cc:1261] T a58adae0b7574cef8c19e5b2b0781603 P 782e9d502c824b62af4770b0b466cc2a [term 1 FOLLOWER]: Refusing update from remote peer d23fa4b8d26a4a87bcb9817236b7f1a2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 21:11:41.806432 19992 consensus_queue.cc:1035] T a58adae0b7574cef8c19e5b2b0781603 P d23fa4b8d26a4a87bcb9817236b7f1a2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "782e9d502c824b62af4770b0b466cc2a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 21:11:41.806535 19989 consensus_queue.cc:1035] T a58adae0b7574cef8c19e5b2b0781603 P d23fa4b8d26a4a87bcb9817236b7f1a2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10e88b55c0c34968b9e1ea6fe49ac885" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41403 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 21:11:41.808846 19995 mvcc.cc:205] Tried to move back new op lower bound from 7333004500194955264 to 7333004500170768384. Current Snapshot: MvccSnapshot[applied={T|T < 7333004500194955264}]
I20260924 21:11:41.808949 19997 mvcc.cc:205] Tried to move back new op lower bound from 7333004500194955264 to 7333004500170768384. Current Snapshot: MvccSnapshot[applied={T|T < 7333004500194955264}]
I20260924 21:11:41.809525 19996 mvcc.cc:205] Tried to move back new op lower bound from 7333004500194955264 to 7333004500170768384. Current Snapshot: MvccSnapshot[applied={T|T < 7333004500194955264 or (T in {7333004500194955264})}]
I20260924 21:11:41.811411 19509 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40372:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 21:11:41.811503 19509 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40372:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 21:11:41.812145 19509 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 54b15ed90a9f4be5a6585de359f0af12: Sending DeleteTablet for 3 replicas of tablet a58adae0b7574cef8c19e5b2b0781603
I20260924 21:11:41.812429 19634 tablet_service.cc:1514] Processing DeleteTablet for tablet a58adae0b7574cef8c19e5b2b0781603 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 21:11:41 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:35710
I20260924 21:11:41.812516 19902 tablet_service.cc:1514] Processing DeleteTablet for tablet a58adae0b7574cef8c19e5b2b0781603 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 21:11:41 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:56242
I20260924 21:11:41.812620 20005 tablet_replica.cc:330] T a58adae0b7574cef8c19e5b2b0781603 P d23fa4b8d26a4a87bcb9817236b7f1a2: stopping tablet replica
I20260924 21:11:41.812700 20005 raft_consensus.cc:2229] T a58adae0b7574cef8c19e5b2b0781603 P d23fa4b8d26a4a87bcb9817236b7f1a2 [term 1 LEADER]: Raft consensus shutting down.
I20260924 21:11:41.812691 19768 tablet_service.cc:1514] Processing DeleteTablet for tablet a58adae0b7574cef8c19e5b2b0781603 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 21:11:41 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:54574
I20260924 21:11:41.812829 20005 raft_consensus.cc:2258] T a58adae0b7574cef8c19e5b2b0781603 P d23fa4b8d26a4a87bcb9817236b7f1a2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 21:11:41.813009 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAzNNjS/build/release/bin/kudu with pid 19571
W20260924 21:11:41.817873 19494 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:45995 (error 108)
I20260924 21:11:41.817914 20006 tablet_replica.cc:330] T a58adae0b7574cef8c19e5b2b0781603 P 782e9d502c824b62af4770b0b466cc2a: stopping tablet replica
I20260924 21:11:41.817965 20007 tablet_replica.cc:330] T a58adae0b7574cef8c19e5b2b0781603 P 10e88b55c0c34968b9e1ea6fe49ac885: stopping tablet replica
I20260924 21:11:41.817984 20006 raft_consensus.cc:2229] T a58adae0b7574cef8c19e5b2b0781603 P 782e9d502c824b62af4770b0b466cc2a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 21:11:41.818051 20007 raft_consensus.cc:2229] T a58adae0b7574cef8c19e5b2b0781603 P 10e88b55c0c34968b9e1ea6fe49ac885 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 21:11:41.818080 20006 raft_consensus.cc:2258] T a58adae0b7574cef8c19e5b2b0781603 P 782e9d502c824b62af4770b0b466cc2a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 21:11:41.818243 20007 raft_consensus.cc:2258] T a58adae0b7574cef8c19e5b2b0781603 P 10e88b55c0c34968b9e1ea6fe49ac885 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 21:11:41.818394 19980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:11:41.818272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45995 (local address 127.0.0.1:35738)
0924 21:11:41.818325 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0924 21:11:41.818363 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:45995: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260924 21:11:41.818508 19494 catalog_manager.cc:4584] TS d23fa4b8d26a4a87bcb9817236b7f1a2 (127.0.98.1:45995): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet a58adae0b7574cef8c19e5b2b0781603: Network error: Client connection negotiation failed: client connection to 127.0.98.1:45995: connect: Connection refused (error 111)
I20260924 21:11:41.818557 20006 ts_tablet_manager.cc:1905] T a58adae0b7574cef8c19e5b2b0781603 P 782e9d502c824b62af4770b0b466cc2a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 21:11:41.818706 20007 ts_tablet_manager.cc:1905] T a58adae0b7574cef8c19e5b2b0781603 P 10e88b55c0c34968b9e1ea6fe49ac885: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 21:11:41.818800 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAzNNjS/build/release/bin/kudu with pid 19705
I20260924 21:11:41.821755 20006 ts_tablet_manager.cc:1918] T a58adae0b7574cef8c19e5b2b0781603 P 782e9d502c824b62af4770b0b466cc2a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 21:11:41.821849 20006 log.cc:1196] T a58adae0b7574cef8c19e5b2b0781603 P 782e9d502c824b62af4770b0b466cc2a: Deleting WAL directory at /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790284076800039-392-0/minicluster-data/ts-2/wal/wals/a58adae0b7574cef8c19e5b2b0781603
I20260924 21:11:41.822103 20006 ts_tablet_manager.cc:1939] T a58adae0b7574cef8c19e5b2b0781603 P 782e9d502c824b62af4770b0b466cc2a: Deleting consensus metadata
I20260924 21:11:41.822439 19495 catalog_manager.cc:4841] TS 782e9d502c824b62af4770b0b466cc2a (127.0.98.3:44861): tablet a58adae0b7574cef8c19e5b2b0781603 (table test-table [id=74389c9843a14932897d443e8b7dc395]) successfully deleted
I20260924 21:11:41.823199 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAzNNjS/build/release/bin/kudu with pid 19839
I20260924 21:11:41.823210 19980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:11:41.823101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41403 (local address 127.0.0.1:54590)
0924 21:11:41.823157 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0924 21:11:41.823184 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:41403: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
I20260924 21:11:41.827838 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAzNNjS/build/release/bin/kudu with pid 19477
2026-09-24T21:11:41Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (1816 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-24T21:11:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T21:11:41Z Disabled control of system clock
I20260924 21:11:41.853363 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAzNNjS/build/release/bin/kudu
/tmp/dist-test-taskAzNNjS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790284076800039-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790284076800039-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790284076800039-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790284076800039-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45669
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:40763
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45669
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 21:11:41.923513 20012 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 21:11:41.923708 20012 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 21:11:41.923728 20012 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 21:11:41.924688 20012 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 21:11:41.924717 20012 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 21:11:41.924729 20012 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 21:11:41.924741 20012 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 21:11:41.924751 20012 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 21:11:41.925890 20012 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40763
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790284076800039-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790284076800039-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45669
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45669
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790284076800039-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20012
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790284076800039-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 19:49:06 UTC on ccycloud.quasar-vhqjoc.root.comops.site
build id 4670864
I20260924 21:11:41.926043 20012 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 21:11:41.926223 20012 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 21:11:41.930356 20018 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 21:11:41.930313 20020 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 21:11:41.930343 20017 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 21:11:41.930523 20012 server_base.cc:1034] running on GCE node
I20260924 21:11:41.930744 20012 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 21:11:41.930992 20012 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 21:11:41.932133 20012 hybrid_clock.cc:648] HybridClock initialized: now 1790284301932125 us; error 28 us; skew 500 ppm
I20260924 21:11:41.933138 20012 webserver.cc:460] Webserver started at http://127.0.98.62:37001/ using document root <none> and password file <none>
I20260924 21:11:41.933360 20012 fs_manager.cc:359] Metadata directory not provided
I20260924 21:11:41.933401 20012 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 21:11:41.933535 20012 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 21:11:41.934552 20012 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790284076800039-392-0/minicluster-data/master-0/data/instance:
uuid: "c459f6c1ce8949fb9a392c55178fbdd2"
format_stamp: "Formatted at 2026-09-24 21:11:41 on dist-test-slave-c7tk"
server_key: "af1cee65091bf4b3c5a037196c097b54"
server_key_iv: "69bed7ded08a3c8ed18e3e4faef864ff"
server_key_version: "encryptionkey@0"
I20260924 21:11:41.934859 20012 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790284076800039-392-0/minicluster-data/master-0/wal/instance:
uuid: "c459f6c1ce8949fb9a392c55178fbdd2"
format_stamp: "Formatted at 2026-09-24 21:11:41 on dist-test-slave-c7tk"
server_key: "af1cee65091bf4b3c5a037196c097b54"
server_key_iv: "69bed7ded08a3c8ed18e3e4faef864ff"
server_key_version: "encryptionkey@0"
I20260924 21:11:41.935959 20012 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260924 21:11:41.936586 20026 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 21:11:41.936734 20012 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 21:11:41.936816 20012 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790284076800039-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790284076800039-392-0/minicluster-data/master-0/wal
uuid: "c459f6c1ce8949fb9a392c55178fbdd2"
format_stamp: "Formatted at 2026-09-24 21:11:41 on dist-test-slave-c7tk"
server_key: "af1cee65091bf4b3c5a037196c097b54"
server_key_iv: "69bed7ded08a3c8ed18e3e4faef864ff"
server_key_version: "encryptionkey@0"
I20260924 21:11:41.936890 20012 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790284076800039-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790284076800039-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790284076800039-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 21:11:41.947031 20012 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 21:11:41.947284 20012 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 21:11:41.947408 20012 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 21:11:41.950794 20012 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45669
I20260924 21:11:41.950821 20078 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45669 every 8 connection(s)
I20260924 21:11:41.951090 20012 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790284076800039-392-0/minicluster-data/master-0/data/info.pb
I20260924 21:11:41.951884 20079 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 21:11:41.953717 20079 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c459f6c1ce8949fb9a392c55178fbdd2: Bootstrap starting.
I20260924 21:11:41.954258 20079 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c459f6c1ce8949fb9a392c55178fbdd2: Neither blocks nor log segments found. Creating new log.
I20260924 21:11:41.954546 20079 log.cc:824] T 00000000000000000000000000000000 P c459f6c1ce8949fb9a392c55178fbdd2: Log is configured to *not* fsync() on all Append() calls
I20260924 21:11:41.955080 20079 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c459f6c1ce8949fb9a392c55178fbdd2: No bootstrap required, opened a new log
I20260924 21:11:41.956362 20079 raft_consensus.cc:348] T 00000000000000000000000000000000 P c459f6c1ce8949fb9a392c55178fbdd2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c459f6c1ce8949fb9a392c55178fbdd2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45669 } }
I20260924 21:11:41.956467 20079 raft_consensus.cc:374] T 00000000000000000000000000000000 P c459f6c1ce8949fb9a392c55178fbdd2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 21:11:41.956517 20079 raft_consensus.cc:729] T 00000000000000000000000000000000 P c459f6c1ce8949fb9a392c55178fbdd2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c459f6c1ce8949fb9a392c55178fbdd2, State: Initialized, Role: FOLLOWER
I20260924 21:11:41.956651 20079 consensus_queue.cc:260] T 00000000000000000000000000000000 P c459f6c1ce8949fb9a392c55178fbdd2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c459f6c1ce8949fb9a392c55178fbdd2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45669 } }
I20260924 21:11:41.956707 20079 raft_consensus.cc:388] T 00000000000000000000000000000000 P c459f6c1ce8949fb9a392c55178fbdd2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 21:11:41.956750 20079 raft_consensus.cc:482] T 00000000000000000000000000000000 P c459f6c1ce8949fb9a392c55178fbdd2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 21:11:41.956792 20079 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c459f6c1ce8949fb9a392c55178fbdd2 [term 0 FOLLOWER]: Advancing to term 1
I20260924 21:11:41.956789 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAzNNjS/build/release/bin/kudu as pid 20012
I20260924 21:11:41.956874 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790284076800039-392-0/minicluster-data/master-0/wal/instance
I20260924 21:11:41.957028 392 external_mini_cluster.cc:1428] Setting key 8536c44f2331de99ef8a1d334623517e
I20260924 21:11:41.957309 20079 raft_consensus.cc:504] T 00000000000000000000000000000000 P c459f6c1ce8949fb9a392c55178fbdd2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c459f6c1ce8949fb9a392c55178fbdd2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45669 } }
I20260924 21:11:41.957405 20079 leader_election.cc:302] T 00000000000000000000000000000000 P c459f6c1ce8949fb9a392c55178fbdd2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c459f6c1ce8949fb9a392c55178fbdd2; no voters:
I20260924 21:11:41.957554 20079 leader_election.cc:288] T 00000000000000000000000000000000 P c459f6c1ce8949fb9a392c55178fbdd2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 21:11:41.957605 20084 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c459f6c1ce8949fb9a392c55178fbdd2 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 21:11:41.957726 20084 raft_consensus.cc:686] T 00000000000000000000000000000000 P c459f6c1ce8949fb9a392c55178fbdd2 [term 1 LEADER]: Becoming Leader. State: Replica: c459f6c1ce8949fb9a392c55178fbdd2, State: Running, Role: LEADER
I20260924 21:11:41.957837 20079 sys_catalog.cc:564] T 00000000000000000000000000000000 P c459f6c1ce8949fb9a392c55178fbdd2 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 21:11:41.957819 20084 consensus_queue.cc:237] T 00000000000000000000000000000000 P c459f6c1ce8949fb9a392c55178fbdd2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c459f6c1ce8949fb9a392c55178fbdd2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45669 } }
I20260924 21:11:41.958179 20085 sys_catalog.cc:455] T 00000000000000000000000000000000 P c459f6c1ce8949fb9a392c55178fbdd2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c459f6c1ce8949fb9a392c55178fbdd2. Latest consensus state: current_term: 1 leader_uuid: "c459f6c1ce8949fb9a392c55178fbdd2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c459f6c1ce8949fb9a392c55178fbdd2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45669 } } }
I20260924 21:11:41.958314 20086 sys_catalog.cc:455] T 00000000000000000000000000000000 P c459f6c1ce8949fb9a392c55178fbdd2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c459f6c1ce8949fb9a392c55178fbdd2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c459f6c1ce8949fb9a392c55178fbdd2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45669 } } }
I20260924 21:11:41.958382 20085 sys_catalog.cc:458] T 00000000000000000000000000000000 P c459f6c1ce8949fb9a392c55178fbdd2 [sys.catalog]: This master's current role is: LEADER
I20260924 21:11:41.958393 20086 sys_catalog.cc:458] T 00000000000000000000000000000000 P c459f6c1ce8949fb9a392c55178fbdd2 [sys.catalog]: This master's current role is: LEADER
I20260924 21:11:41.958700 20093 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 21:11:41.959388 20093 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 21:11:41.960814 20093 catalog_manager.cc:1347] Generated new cluster ID: a372edcbcba1415881e52dab3f4a1623
I20260924 21:11:41.960860 20093 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 21:11:41.962703 20083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:11:41.957405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35250 (local address 127.0.98.62:45669)
0924 21:11:41.957666 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0924 21:11:41.957674 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 21:11:41.957704 (+ 30us) server_negotiation.cc:407] Connection header received
0924 21:11:41.957839 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:11:41.957845 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:11:41.957906 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:11:41.958307 (+ 401us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:11:41.958407 (+ 100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:11:41.959866 (+ 1459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:11:41.960765 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:11:41.961257 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:11:41.961359 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:11:41.961939 (+ 580us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 21:11:41.961943 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 21:11:41.961943 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 21:11:41.962120 (+ 177us) server_negotiation.cc:922] calling sasl_server_start()
0924 21:11:41.962155 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 21:11:41.962168 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 21:11:41.962428 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0924 21:11:41.962454 (+ 26us) server_negotiation.cc:299] Negotiation successful
0924 21:11:41.962521 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":195,"thread_start_us":73,"threads_started":1}
I20260924 21:11:41.976176 20093 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 21:11:41.976759 20093 catalog_manager.cc:1504] Loading token signing keys...
I20260924 21:11:41.980020 20093 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c459f6c1ce8949fb9a392c55178fbdd2: Generated new TSK 0
I20260924 21:11:41.980170 20093 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 21:11:41.981535 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAzNNjS/build/release/bin/kudu
/tmp/dist-test-taskAzNNjS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790284076800039-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790284076800039-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790284076800039-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790284076800039-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45669
--builtin_ntp_servers=127.0.98.20:40763
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 21:11:42.064157 20103 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 21:11:42.064314 20103 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 21:11:42.064332 20103 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 21:11:42.065274 20103 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 21:11:42.065306 20103 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 21:11:42.065327 20103 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 21:11:42.066495 20103 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40763
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790284076800039-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790284076800039-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790284076800039-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45669
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20103
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790284076800039-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 19:49:06 UTC on ccycloud.quasar-vhqjoc.root.comops.site
build id 4670864
I20260924 21:11:42.066700 20103 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 21:11:42.066941 20103 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 21:11:42.072222 20108 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 21:11:42.072346 20111 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 21:11:42.072284 20109 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 21:11:42.072467 20103 server_base.cc:1034] running on GCE node
I20260924 21:11:42.072645 20103 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 21:11:42.072872 20103 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 21:11:42.074023 20103 hybrid_clock.cc:648] HybridClock initialized: now 1790284302074012 us; error 27 us; skew 500 ppm
I20260924 21:11:42.075152 20103 webserver.cc:460] Webserver started at http://127.0.98.1:38329/ using document root <none> and password file <none>
I20260924 21:11:42.075362 20103 fs_manager.cc:359] Metadata directory not provided
I20260924 21:11:42.075423 20103 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 21:11:42.075553 20103 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 21:11:42.076463 20103 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790284076800039-392-0/minicluster-data/ts-0/data/instance:
uuid: "ce2fa930bc8b4badbfda1890515be8fb"
format_stamp: "Formatted at 2026-09-24 21:11:42 on dist-test-slave-c7tk"
server_key: "d5a509bdae36f756f753764820511a5b"
server_key_iv: "e5c00d3c105363df29096ac634da8041"
server_key_version: "encryptionkey@0"
I20260924 21:11:42.076781 20103 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790284076800039-392-0/minicluster-data/ts-0/wal/instance:
uuid: "ce2fa930bc8b4badbfda1890515be8fb"
format_stamp: "Formatted at 2026-09-24 21:11:42 on dist-test-slave-c7tk"
server_key: "d5a509bdae36f756f753764820511a5b"
server_key_iv: "e5c00d3c105363df29096ac634da8041"
server_key_version: "encryptionkey@0"
I20260924 21:11:42.077965 20103 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260924 21:11:42.078634 20117 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 21:11:42.078806 20103 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 21:11:42.078909 20103 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790284076800039-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790284076800039-392-0/minicluster-data/ts-0/wal
uuid: "ce2fa930bc8b4badbfda1890515be8fb"
format_stamp: "Formatted at 2026-09-24 21:11:42 on dist-test-slave-c7tk"
server_key: "d5a509bdae36f756f753764820511a5b"
server_key_iv: "e5c00d3c105363df29096ac634da8041"
server_key_version: "encryptionkey@0"
I20260924 21:11:42.078992 20103 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790284076800039-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790284076800039-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790284076800039-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 21:11:42.085039 20103 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 21:11:42.085280 20103 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 21:11:42.085404 20103 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 21:11:42.085623 20103 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 21:11:42.085939 20103 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 21:11:42.085985 20103 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 21:11:42.086017 20103 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 21:11:42.086040 20103 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 21:11:42.091310 20103 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:41809
I20260924 21:11:42.091341 20230 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:41809 every 8 connection(s)
I20260924 21:11:42.091605 20103 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790284076800039-392-0/minicluster-data/ts-0/data/info.pb
I20260924 21:11:42.095813 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAzNNjS/build/release/bin/kudu as pid 20103
I20260924 21:11:42.095908 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790284076800039-392-0/minicluster-data/ts-0/wal/instance
I20260924 21:11:42.096098 392 external_mini_cluster.cc:1428] Setting key ff8f2397841cdd7cdd795c620a7b3071
I20260924 21:11:42.096108 20083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:11:42.092226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:50527 (local address 127.0.98.62:45669)
0924 21:11:42.092298 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0924 21:11:42.092301 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 21:11:42.092465 (+ 164us) server_negotiation.cc:407] Connection header received
0924 21:11:42.092867 (+ 402us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:11:42.092868 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:11:42.092878 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:11:42.092925 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:11:42.093463 (+ 538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:11:42.093975 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:11:42.094909 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:11:42.095112 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:11:42.095157 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:11:42.095811 (+ 654us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 21:11:42.095813 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 21:11:42.095813 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 21:11:42.095819 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 21:11:42.095835 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 21:11:42.095839 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 21:11:42.095898 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0924 21:11:42.096020 (+ 122us) server_negotiation.cc:299] Negotiation successful
0924 21:11:42.096052 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260924 21:11:42.096483 20233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:11:42.092207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45669 (local address 127.0.98.1:50527)
0924 21:11:42.092367 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0924 21:11:42.092394 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0924 21:11:42.092791 (+ 397us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 21:11:42.092968 (+ 177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 21:11:42.092970 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 21:11:42.092977 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0924 21:11:42.093399 (+ 422us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:11:42.093402 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:11:42.094045 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:11:42.094046 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:11:42.094827 (+ 781us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:11:42.094829 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:11:42.095194 (+ 365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:11:42.095195 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:11:42.095282 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:11:42.095690 (+ 408us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 21:11:42.095690 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0924 21:11:42.095712 (+ 22us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 21:11:42.095729 (+ 17us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 21:11:42.095901 (+ 172us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 21:11:42.095904 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 21:11:42.095907 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 21:11:42.096211 (+ 304us) client_negotiation.cc:240] Negotiation successful
0924 21:11:42.096319 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":94,"threads_started":1}
I20260924 21:11:42.096902 20231 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45669
I20260924 21:11:42.096979 20231 heartbeater.cc:464] Registering TS with master...
I20260924 21:11:42.097056 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAzNNjS/build/release/bin/kudu
/tmp/dist-test-taskAzNNjS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790284076800039-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790284076800039-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790284076800039-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790284076800039-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45669
--builtin_ntp_servers=127.0.98.20:40763
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260924 21:11:42.097149 20231 heartbeater.cc:511] Master 127.0.98.62:45669 requested a full tablet report, sending...
I20260924 21:11:42.097554 20043 ts_manager.cc:194] Registered new tserver with Master: ce2fa930bc8b4badbfda1890515be8fb (127.0.98.1:41809)
I20260924 21:11:42.098239 20043 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:50527
W20260924 21:11:42.167757 20234 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 21:11:42.167906 20234 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 21:11:42.167924 20234 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 21:11:42.168887 20234 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 21:11:42.168919 20234 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 21:11:42.168943 20234 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 21:11:42.170161 20234 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40763
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790284076800039-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790284076800039-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790284076800039-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45669
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20234
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790284076800039-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 19:49:06 UTC on ccycloud.quasar-vhqjoc.root.comops.site
build id 4670864
I20260924 21:11:42.170303 20234 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 21:11:42.170488 20234 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 21:11:42.174746 20239 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 21:11:42.174758 20240 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 21:11:42.174734 20242 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 21:11:42.174945 20234 server_base.cc:1034] running on GCE node
I20260924 21:11:42.175182 20234 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 21:11:42.175347 20234 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 21:11:42.176487 20234 hybrid_clock.cc:648] HybridClock initialized: now 1790284302176476 us; error 26 us; skew 500 ppm
I20260924 21:11:42.177691 20234 webserver.cc:460] Webserver started at http://127.0.98.2:35261/ using document root <none> and password file <none>
I20260924 21:11:42.177915 20234 fs_manager.cc:359] Metadata directory not provided
I20260924 21:11:42.177959 20234 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 21:11:42.178090 20234 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 21:11:42.179059 20234 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790284076800039-392-0/minicluster-data/ts-1/data/instance:
uuid: "80fb8f02fad44a87a650e8a715951ab4"
format_stamp: "Formatted at 2026-09-24 21:11:42 on dist-test-slave-c7tk"
server_key: "29f290228c19a3e406d48780aa35b8ec"
server_key_iv: "b6135f07216bbb3eca7daae61e002113"
server_key_version: "encryptionkey@0"
I20260924 21:11:42.179385 20234 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790284076800039-392-0/minicluster-data/ts-1/wal/instance:
uuid: "80fb8f02fad44a87a650e8a715951ab4"
format_stamp: "Formatted at 2026-09-24 21:11:42 on dist-test-slave-c7tk"
server_key: "29f290228c19a3e406d48780aa35b8ec"
server_key_iv: "b6135f07216bbb3eca7daae61e002113"
server_key_version: "encryptionkey@0"
I20260924 21:11:42.180552 20234 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 21:11:42.181252 20248 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 21:11:42.181399 20234 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 21:11:42.181483 20234 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790284076800039-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790284076800039-392-0/minicluster-data/ts-1/wal
uuid: "80fb8f02fad44a87a650e8a715951ab4"
format_stamp: "Formatted at 2026-09-24 21:11:42 on dist-test-slave-c7tk"
server_key: "29f290228c19a3e406d48780aa35b8ec"
server_key_iv: "b6135f07216bbb3eca7daae61e002113"
server_key_version: "encryptionkey@0"
I20260924 21:11:42.181560 20234 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790284076800039-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790284076800039-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790284076800039-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 21:11:42.198848 20234 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 21:11:42.199102 20234 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 21:11:42.199244 20234 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 21:11:42.199466 20234 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 21:11:42.199805 20234 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 21:11:42.199859 20234 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 21:11:42.199893 20234 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 21:11:42.199918 20234 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 21:11:42.204994 20234 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:41073
I20260924 21:11:42.205026 20361 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:41073 every 8 connection(s)
I20260924 21:11:42.205284 20234 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790284076800039-392-0/minicluster-data/ts-1/data/info.pb
I20260924 21:11:42.209450 20083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:11:42.205875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:43029 (local address 127.0.98.62:45669)
0924 21:11:42.205942 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0924 21:11:42.205945 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 21:11:42.206139 (+ 194us) server_negotiation.cc:407] Connection header received
0924 21:11:42.206528 (+ 389us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:11:42.206529 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:11:42.206537 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:11:42.206608 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:11:42.207132 (+ 524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:11:42.207545 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:11:42.208291 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:11:42.208438 (+ 147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:11:42.208482 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:11:42.209268 (+ 786us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 21:11:42.209269 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 21:11:42.209269 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 21:11:42.209273 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 21:11:42.209282 (+ 9us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 21:11:42.209285 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 21:11:42.209322 (+ 37us) server_negotiation.cc:1035] Waiting for connection context
0924 21:11:42.209380 (+ 58us) server_negotiation.cc:299] Negotiation successful
0924 21:11:42.209398 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260924 21:11:42.209664 20364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:11:42.205863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45669 (local address 127.0.98.2:43029)
0924 21:11:42.206040 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0924 21:11:42.206067 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0924 21:11:42.206460 (+ 393us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 21:11:42.206652 (+ 192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 21:11:42.206655 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 21:11:42.206661 (+ 6us) client_negotiation.cc:189] Negotiated authn=SASL
0924 21:11:42.207072 (+ 411us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:11:42.207076 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:11:42.207608 (+ 532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:11:42.207608 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:11:42.208231 (+ 623us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:11:42.208233 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:11:42.208520 (+ 287us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:11:42.208521 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:11:42.208636 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:11:42.209157 (+ 521us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 21:11:42.209157 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0924 21:11:42.209182 (+ 25us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 21:11:42.209201 (+ 19us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 21:11:42.209322 (+ 121us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 21:11:42.209324 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 21:11:42.209326 (+ 2us) client_negotiation.cc:769] Sending connection context
0924 21:11:42.209370 (+ 44us) client_negotiation.cc:240] Negotiation successful
0924 21:11:42.209435 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":88,"threads_started":1}
I20260924 21:11:42.210158 20362 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45669
I20260924 21:11:42.210254 20362 heartbeater.cc:464] Registering TS with master...
I20260924 21:11:42.210412 20362 heartbeater.cc:511] Master 127.0.98.62:45669 requested a full tablet report, sending...
I20260924 21:11:42.210683 20043 ts_manager.cc:194] Registered new tserver with Master: 80fb8f02fad44a87a650e8a715951ab4 (127.0.98.2:41073)
I20260924 21:11:42.210949 20043 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:43029
I20260924 21:11:42.211495 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAzNNjS/build/release/bin/kudu as pid 20234
I20260924 21:11:42.211606 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790284076800039-392-0/minicluster-data/ts-1/wal/instance
I20260924 21:11:42.211742 392 external_mini_cluster.cc:1428] Setting key 03d8ba08a63389ce2cfeadaa801f92c6
I20260924 21:11:42.212420 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAzNNjS/build/release/bin/kudu
/tmp/dist-test-taskAzNNjS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790284076800039-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790284076800039-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790284076800039-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790284076800039-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45669
--builtin_ntp_servers=127.0.98.20:40763
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 21:11:42.283864 20365 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 21:11:42.284021 20365 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 21:11:42.284039 20365 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 21:11:42.285115 20365 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 21:11:42.285148 20365 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 21:11:42.285168 20365 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 21:11:42.286293 20365 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40763
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790284076800039-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790284076800039-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790284076800039-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45669
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20365
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790284076800039-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 19:49:06 UTC on ccycloud.quasar-vhqjoc.root.comops.site
build id 4670864
I20260924 21:11:42.286437 20365 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 21:11:42.286621 20365 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 21:11:42.290844 20370 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 21:11:42.290830 20373 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 21:11:42.290855 20371 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 21:11:42.290940 20365 server_base.cc:1034] running on GCE node
I20260924 21:11:42.291162 20365 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 21:11:42.291409 20365 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 21:11:42.292569 20365 hybrid_clock.cc:648] HybridClock initialized: now 1790284302292549 us; error 42 us; skew 500 ppm
I20260924 21:11:42.293692 20365 webserver.cc:460] Webserver started at http://127.0.98.3:42355/ using document root <none> and password file <none>
I20260924 21:11:42.293949 20365 fs_manager.cc:359] Metadata directory not provided
I20260924 21:11:42.293989 20365 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 21:11:42.294140 20365 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 21:11:42.295037 20365 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790284076800039-392-0/minicluster-data/ts-2/data/instance:
uuid: "2d5d3b999a0d40a8a3e514f41fd51cdf"
format_stamp: "Formatted at 2026-09-24 21:11:42 on dist-test-slave-c7tk"
server_key: "1b86f7f6d0bc010e0c1a35942c0ef49d"
server_key_iv: "1c6cce7f4ca2c8580f211e1350c169b9"
server_key_version: "encryptionkey@0"
I20260924 21:11:42.295363 20365 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790284076800039-392-0/minicluster-data/ts-2/wal/instance:
uuid: "2d5d3b999a0d40a8a3e514f41fd51cdf"
format_stamp: "Formatted at 2026-09-24 21:11:42 on dist-test-slave-c7tk"
server_key: "1b86f7f6d0bc010e0c1a35942c0ef49d"
server_key_iv: "1c6cce7f4ca2c8580f211e1350c169b9"
server_key_version: "encryptionkey@0"
I20260924 21:11:42.296458 20365 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 21:11:42.297299 20379 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 21:11:42.297493 20365 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 21:11:42.297573 20365 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790284076800039-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790284076800039-392-0/minicluster-data/ts-2/wal
uuid: "2d5d3b999a0d40a8a3e514f41fd51cdf"
format_stamp: "Formatted at 2026-09-24 21:11:42 on dist-test-slave-c7tk"
server_key: "1b86f7f6d0bc010e0c1a35942c0ef49d"
server_key_iv: "1c6cce7f4ca2c8580f211e1350c169b9"
server_key_version: "encryptionkey@0"
I20260924 21:11:42.297631 20365 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790284076800039-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790284076800039-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790284076800039-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 21:11:42.311775 20365 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 21:11:42.311949 20365 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 21:11:42.312026 20365 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 21:11:42.312166 20365 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 21:11:42.312367 20365 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 21:11:42.312391 20365 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 21:11:42.312408 20365 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 21:11:42.312419 20365 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 21:11:42.317623 20365 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:36341
I20260924 21:11:42.317675 20492 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:36341 every 8 connection(s)
I20260924 21:11:42.317989 20365 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790284076800039-392-0/minicluster-data/ts-2/data/info.pb
I20260924 21:11:42.322366 20083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:11:42.318678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:44555 (local address 127.0.98.62:45669)
0924 21:11:42.318718 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0924 21:11:42.318721 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 21:11:42.318929 (+ 208us) server_negotiation.cc:407] Connection header received
0924 21:11:42.319376 (+ 447us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:11:42.319377 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:11:42.319386 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:11:42.319436 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:11:42.320028 (+ 592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:11:42.320488 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:11:42.321300 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:11:42.321465 (+ 165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:11:42.321509 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:11:42.322160 (+ 651us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 21:11:42.322161 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 21:11:42.322161 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 21:11:42.322165 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 21:11:42.322178 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 21:11:42.322181 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 21:11:42.322220 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0924 21:11:42.322291 (+ 71us) server_negotiation.cc:299] Negotiation successful
0924 21:11:42.322315 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260924 21:11:42.322520 20495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:11:42.318631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45669 (local address 127.0.98.3:44555)
0924 21:11:42.318808 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0924 21:11:42.318844 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0924 21:11:42.319300 (+ 456us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 21:11:42.319495 (+ 195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 21:11:42.319498 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 21:11:42.319505 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0924 21:11:42.319969 (+ 464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:11:42.319972 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:11:42.320560 (+ 588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:11:42.320561 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:11:42.321233 (+ 672us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:11:42.321235 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:11:42.321538 (+ 303us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:11:42.321538 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:11:42.321628 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:11:42.322038 (+ 410us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 21:11:42.322038 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0924 21:11:42.322062 (+ 24us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 21:11:42.322080 (+ 18us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 21:11:42.322230 (+ 150us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 21:11:42.322234 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 21:11:42.322236 (+ 2us) client_negotiation.cc:769] Sending connection context
0924 21:11:42.322281 (+ 45us) client_negotiation.cc:240] Negotiation successful
0924 21:11:42.322346 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":90,"threads_started":1}
I20260924 21:11:42.322969 20493 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45669
I20260924 21:11:42.323067 20493 heartbeater.cc:464] Registering TS with master...
I20260924 21:11:42.323221 20493 heartbeater.cc:511] Master 127.0.98.62:45669 requested a full tablet report, sending...
I20260924 21:11:42.323561 20043 ts_manager.cc:194] Registered new tserver with Master: 2d5d3b999a0d40a8a3e514f41fd51cdf (127.0.98.3:36341)
I20260924 21:11:42.323904 20043 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:44555
I20260924 21:11:42.326524 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAzNNjS/build/release/bin/kudu as pid 20365
I20260924 21:11:42.326604 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790284076800039-392-0/minicluster-data/ts-2/wal/instance
I20260924 21:11:42.326730 392 external_mini_cluster.cc:1428] Setting key 31acdddcfa962b2426301fbe0624deb7
I20260924 21:11:42.327550 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 21:11:42.331167 20083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:11:42.328839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35256 (local address 127.0.98.62:45669)
0924 21:11:42.328893 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0924 21:11:42.328895 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 21:11:42.328990 (+ 95us) server_negotiation.cc:407] Connection header received
0924 21:11:42.329042 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:11:42.329043 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:11:42.329052 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:11:42.329103 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:11:42.329265 (+ 162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:11:42.329688 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:11:42.330388 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:11:42.330541 (+ 153us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:11:42.330587 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:11:42.331019 (+ 432us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 21:11:42.331020 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 21:11:42.331021 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 21:11:42.331024 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0924 21:11:42.331032 (+ 8us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 21:11:42.331035 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 21:11:42.331075 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0924 21:11:42.331108 (+ 33us) server_negotiation.cc:299] Negotiation successful
0924 21:11:42.331124 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260924 21:11:42.332604 20043 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35256:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 21:11:42.334026 20043 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 21:11:42.339745 20501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:11:42.335914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41809 (local address 127.0.0.1:51380)
0924 21:11:42.336174 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0924 21:11:42.336183 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 21:11:42.336219 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 21:11:42.336382 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 21:11:42.336384 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 21:11:42.336387 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 21:11:42.336439 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:11:42.336441 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:11:42.337622 (+ 1181us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:11:42.337623 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:11:42.338479 (+ 856us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:11:42.338482 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:11:42.339447 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:11:42.339448 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:11:42.339499 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:11:42.339508 (+ 9us) client_negotiation.cc:769] Sending connection context
0924 21:11:42.339597 (+ 89us) client_negotiation.cc:240] Negotiation successful
0924 21:11:42.339647 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250,"mutex_wait_us":43,"thread_start_us":70,"threads_started":1}
I20260924 21:11:42.339757 20504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:11:42.336026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51380 (local address 127.0.98.1:41809)
0924 21:11:42.336203 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0924 21:11:42.336207 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 21:11:42.336218 (+ 11us) server_negotiation.cc:407] Connection header received
0924 21:11:42.336287 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:11:42.336289 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:11:42.336303 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:11:42.336375 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 21:11:42.336495 (+ 120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:11:42.337522 (+ 1027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:11:42.338600 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:11:42.339346 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:11:42.339446 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:11:42.339543 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0924 21:11:42.339610 (+ 67us) server_negotiation.cc:299] Negotiation successful
0924 21:11:42.339645 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":68,"threads_started":1}
I20260924 21:11:42.339967 20502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:11:42.335913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41073 (local address 127.0.0.1:52786)
0924 21:11:42.336064 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0924 21:11:42.336100 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0924 21:11:42.336155 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 21:11:42.336422 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 21:11:42.336423 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 21:11:42.336425 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 21:11:42.336465 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:11:42.336467 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:11:42.337600 (+ 1133us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:11:42.337601 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:11:42.338821 (+ 1220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:11:42.338823 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:11:42.339739 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:11:42.339739 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:11:42.339813 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:11:42.339816 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 21:11:42.339863 (+ 47us) client_negotiation.cc:240] Negotiation successful
0924 21:11:42.339902 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"mutex_wait_us":65,"thread_start_us":54,"threads_started":1}
I20260924 21:11:42.340003 20503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:11:42.335994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52786 (local address 127.0.98.2:41073)
0924 21:11:42.336286 (+ 292us) server_negotiation.cc:206] Beginning negotiation
0924 21:11:42.336289 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 21:11:42.336296 (+ 7us) server_negotiation.cc:407] Connection header received
0924 21:11:42.336332 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:11:42.336334 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:11:42.336347 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:11:42.336428 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 21:11:42.336546 (+ 118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:11:42.337547 (+ 1001us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:11:42.338905 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:11:42.339642 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:11:42.339733 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:11:42.339871 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0924 21:11:42.339894 (+ 23us) server_negotiation.cc:299] Negotiation successful
0924 21:11:42.339925 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":259,"thread_start_us":86,"threads_started":1}
I20260924 21:11:42.340535 20165 tablet_service.cc:1467] Processing CreateTablet for tablet f232d90ec14249e29715dd97b936321b (DEFAULT_TABLE table=test-table [id=c9a56f232f9a47bfaafbaa878bf78889]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 21:11:42.340664 20296 tablet_service.cc:1467] Processing CreateTablet for tablet f232d90ec14249e29715dd97b936321b (DEFAULT_TABLE table=test-table [id=c9a56f232f9a47bfaafbaa878bf78889]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 21:11:42.340873 20165 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f232d90ec14249e29715dd97b936321b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 21:11:42.340998 20296 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f232d90ec14249e29715dd97b936321b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 21:11:42.341315 20505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:11:42.336086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36341 (local address 127.0.0.1:39242)
0924 21:11:42.336509 (+ 423us) negotiation.cc:107] Waiting for socket to connect
0924 21:11:42.336517 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0924 21:11:42.336559 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 21:11:42.336744 (+ 185us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 21:11:42.336745 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 21:11:42.336746 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 21:11:42.336781 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:11:42.336782 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:11:42.337548 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:11:42.337549 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:11:42.338821 (+ 1272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:11:42.338823 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:11:42.341091 (+ 2268us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:11:42.341092 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:11:42.341147 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:11:42.341151 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 21:11:42.341207 (+ 56us) client_negotiation.cc:240] Negotiation successful
0924 21:11:42.341246 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":409,"mutex_wait_us":6,"thread_start_us":53,"threads_started":1}
I20260924 21:11:42.341344 20506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:11:42.336227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39242 (local address 127.0.98.3:36341)
0924 21:11:42.336358 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0924 21:11:42.336362 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 21:11:42.336566 (+ 204us) server_negotiation.cc:407] Connection header received
0924 21:11:42.336618 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:11:42.336620 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:11:42.336633 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:11:42.336695 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 21:11:42.336848 (+ 153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:11:42.337468 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:11:42.340101 (+ 2633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:11:42.340733 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:11:42.340977 (+ 244us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:11:42.341065 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0924 21:11:42.341221 (+ 156us) server_negotiation.cc:299] Negotiation successful
0924 21:11:42.341261 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101,"thread_start_us":72,"threads_started":1}
I20260924 21:11:42.343509 20508 tablet_bootstrap.cc:492] T f232d90ec14249e29715dd97b936321b P ce2fa930bc8b4badbfda1890515be8fb: Bootstrap starting.
I20260924 21:11:42.343921 20427 tablet_service.cc:1467] Processing CreateTablet for tablet f232d90ec14249e29715dd97b936321b (DEFAULT_TABLE table=test-table [id=c9a56f232f9a47bfaafbaa878bf78889]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 21:11:42.344161 20508 tablet_bootstrap.cc:654] T f232d90ec14249e29715dd97b936321b P ce2fa930bc8b4badbfda1890515be8fb: Neither blocks nor log segments found. Creating new log.
I20260924 21:11:42.344201 20427 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f232d90ec14249e29715dd97b936321b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 21:11:42.344444 20508 log.cc:824] T f232d90ec14249e29715dd97b936321b P ce2fa930bc8b4badbfda1890515be8fb: Log is configured to *not* fsync() on all Append() calls
I20260924 21:11:42.344743 20507 tablet_bootstrap.cc:492] T f232d90ec14249e29715dd97b936321b P 80fb8f02fad44a87a650e8a715951ab4: Bootstrap starting.
I20260924 21:11:42.345109 20508 tablet_bootstrap.cc:492] T f232d90ec14249e29715dd97b936321b P ce2fa930bc8b4badbfda1890515be8fb: No bootstrap required, opened a new log
I20260924 21:11:42.345189 20508 ts_tablet_manager.cc:1397] T f232d90ec14249e29715dd97b936321b P ce2fa930bc8b4badbfda1890515be8fb: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 21:11:42.345381 20507 tablet_bootstrap.cc:654] T f232d90ec14249e29715dd97b936321b P 80fb8f02fad44a87a650e8a715951ab4: Neither blocks nor log segments found. Creating new log.
I20260924 21:11:42.345687 20507 log.cc:824] T f232d90ec14249e29715dd97b936321b P 80fb8f02fad44a87a650e8a715951ab4: Log is configured to *not* fsync() on all Append() calls
I20260924 21:11:42.346431 20507 tablet_bootstrap.cc:492] T f232d90ec14249e29715dd97b936321b P 80fb8f02fad44a87a650e8a715951ab4: No bootstrap required, opened a new log
I20260924 21:11:42.346491 20507 ts_tablet_manager.cc:1397] T f232d90ec14249e29715dd97b936321b P 80fb8f02fad44a87a650e8a715951ab4: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 21:11:42.347409 20508 raft_consensus.cc:348] T f232d90ec14249e29715dd97b936321b P ce2fa930bc8b4badbfda1890515be8fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fb8f02fad44a87a650e8a715951ab4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41073 } } peers { permanent_uuid: "ce2fa930bc8b4badbfda1890515be8fb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41809 } } peers { permanent_uuid: "2d5d3b999a0d40a8a3e514f41fd51cdf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36341 } }
I20260924 21:11:42.347548 20511 tablet_bootstrap.cc:492] T f232d90ec14249e29715dd97b936321b P 2d5d3b999a0d40a8a3e514f41fd51cdf: Bootstrap starting.
I20260924 21:11:42.347560 20508 raft_consensus.cc:374] T f232d90ec14249e29715dd97b936321b P ce2fa930bc8b4badbfda1890515be8fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 21:11:42.347601 20508 raft_consensus.cc:729] T f232d90ec14249e29715dd97b936321b P ce2fa930bc8b4badbfda1890515be8fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce2fa930bc8b4badbfda1890515be8fb, State: Initialized, Role: FOLLOWER
I20260924 21:11:42.347702 20508 consensus_queue.cc:260] T f232d90ec14249e29715dd97b936321b P ce2fa930bc8b4badbfda1890515be8fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fb8f02fad44a87a650e8a715951ab4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41073 } } peers { permanent_uuid: "ce2fa930bc8b4badbfda1890515be8fb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41809 } } peers { permanent_uuid: "2d5d3b999a0d40a8a3e514f41fd51cdf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36341 } }
I20260924 21:11:42.347958 20508 ts_tablet_manager.cc:1428] T f232d90ec14249e29715dd97b936321b P ce2fa930bc8b4badbfda1890515be8fb: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260924 21:11:42.347976 20231 heartbeater.cc:503] Master 127.0.98.62:45669 was elected leader, sending a full tablet report...
I20260924 21:11:42.348074 20511 tablet_bootstrap.cc:654] T f232d90ec14249e29715dd97b936321b P 2d5d3b999a0d40a8a3e514f41fd51cdf: Neither blocks nor log segments found. Creating new log.
I20260924 21:11:42.348352 20511 log.cc:824] T f232d90ec14249e29715dd97b936321b P 2d5d3b999a0d40a8a3e514f41fd51cdf: Log is configured to *not* fsync() on all Append() calls
I20260924 21:11:42.348956 20507 raft_consensus.cc:348] T f232d90ec14249e29715dd97b936321b P 80fb8f02fad44a87a650e8a715951ab4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fb8f02fad44a87a650e8a715951ab4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41073 } } peers { permanent_uuid: "ce2fa930bc8b4badbfda1890515be8fb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41809 } } peers { permanent_uuid: "2d5d3b999a0d40a8a3e514f41fd51cdf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36341 } }
I20260924 21:11:42.349086 20511 tablet_bootstrap.cc:492] T f232d90ec14249e29715dd97b936321b P 2d5d3b999a0d40a8a3e514f41fd51cdf: No bootstrap required, opened a new log
I20260924 21:11:42.349119 20507 raft_consensus.cc:374] T f232d90ec14249e29715dd97b936321b P 80fb8f02fad44a87a650e8a715951ab4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 21:11:42.349143 20511 ts_tablet_manager.cc:1397] T f232d90ec14249e29715dd97b936321b P 2d5d3b999a0d40a8a3e514f41fd51cdf: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 21:11:42.349167 20507 raft_consensus.cc:729] T f232d90ec14249e29715dd97b936321b P 80fb8f02fad44a87a650e8a715951ab4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80fb8f02fad44a87a650e8a715951ab4, State: Initialized, Role: FOLLOWER
I20260924 21:11:42.349278 20507 consensus_queue.cc:260] T f232d90ec14249e29715dd97b936321b P 80fb8f02fad44a87a650e8a715951ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fb8f02fad44a87a650e8a715951ab4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41073 } } peers { permanent_uuid: "ce2fa930bc8b4badbfda1890515be8fb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41809 } } peers { permanent_uuid: "2d5d3b999a0d40a8a3e514f41fd51cdf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36341 } }
I20260924 21:11:42.349536 20507 ts_tablet_manager.cc:1428] T f232d90ec14249e29715dd97b936321b P 80fb8f02fad44a87a650e8a715951ab4: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 21:11:42.349591 20362 heartbeater.cc:503] Master 127.0.98.62:45669 was elected leader, sending a full tablet report...
I20260924 21:11:42.351141 20511 raft_consensus.cc:348] T f232d90ec14249e29715dd97b936321b P 2d5d3b999a0d40a8a3e514f41fd51cdf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fb8f02fad44a87a650e8a715951ab4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41073 } } peers { permanent_uuid: "ce2fa930bc8b4badbfda1890515be8fb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41809 } } peers { permanent_uuid: "2d5d3b999a0d40a8a3e514f41fd51cdf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36341 } }
I20260924 21:11:42.351280 20511 raft_consensus.cc:374] T f232d90ec14249e29715dd97b936321b P 2d5d3b999a0d40a8a3e514f41fd51cdf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 21:11:42.351337 20511 raft_consensus.cc:729] T f232d90ec14249e29715dd97b936321b P 2d5d3b999a0d40a8a3e514f41fd51cdf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d5d3b999a0d40a8a3e514f41fd51cdf, State: Initialized, Role: FOLLOWER
I20260924 21:11:42.351434 20511 consensus_queue.cc:260] T f232d90ec14249e29715dd97b936321b P 2d5d3b999a0d40a8a3e514f41fd51cdf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fb8f02fad44a87a650e8a715951ab4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41073 } } peers { permanent_uuid: "ce2fa930bc8b4badbfda1890515be8fb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41809 } } peers { permanent_uuid: "2d5d3b999a0d40a8a3e514f41fd51cdf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36341 } }
I20260924 21:11:42.351655 20511 ts_tablet_manager.cc:1428] T f232d90ec14249e29715dd97b936321b P 2d5d3b999a0d40a8a3e514f41fd51cdf: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260924 21:11:42.351724 20493 heartbeater.cc:503] Master 127.0.98.62:45669 was elected leader, sending a full tablet report...
I20260924 21:11:42.426045 20514 raft_consensus.cc:482] T f232d90ec14249e29715dd97b936321b P 80fb8f02fad44a87a650e8a715951ab4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 21:11:42.426170 20514 raft_consensus.cc:504] T f232d90ec14249e29715dd97b936321b P 80fb8f02fad44a87a650e8a715951ab4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fb8f02fad44a87a650e8a715951ab4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41073 } } peers { permanent_uuid: "ce2fa930bc8b4badbfda1890515be8fb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41809 } } peers { permanent_uuid: "2d5d3b999a0d40a8a3e514f41fd51cdf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36341 } }
I20260924 21:11:42.426626 20514 leader_election.cc:288] T f232d90ec14249e29715dd97b936321b P 80fb8f02fad44a87a650e8a715951ab4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ce2fa930bc8b4badbfda1890515be8fb (127.0.98.1:41809), 2d5d3b999a0d40a8a3e514f41fd51cdf (127.0.98.3:36341)
I20260924 21:11:42.429492 20506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:11:42.426869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:48547 (local address 127.0.98.3:36341)
0924 21:11:42.426912 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0924 21:11:42.426914 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 21:11:42.427045 (+ 131us) server_negotiation.cc:407] Connection header received
0924 21:11:42.427112 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:11:42.427113 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:11:42.427122 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:11:42.427174 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 21:11:42.427337 (+ 163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:11:42.427795 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:11:42.428763 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:11:42.429103 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:11:42.429157 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:11:42.429165 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0924 21:11:42.429363 (+ 198us) server_negotiation.cc:299] Negotiation successful
0924 21:11:42.429418 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 21:11:42.429526 20364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:11:42.426763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41809 (local address 127.0.98.2:52257)
0924 21:11:42.426812 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0924 21:11:42.426822 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 21:11:42.426858 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 21:11:42.427070 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 21:11:42.427070 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 21:11:42.427072 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 21:11:42.427151 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:11:42.427153 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:11:42.427864 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:11:42.427865 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:11:42.428538 (+ 673us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:11:42.428540 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:11:42.429243 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:11:42.429244 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:11:42.429303 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:11:42.429306 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 21:11:42.429378 (+ 72us) client_negotiation.cc:240] Negotiation successful
0924 21:11:42.429440 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29,"mutex_wait_us":27}
I20260924 21:11:42.429514 20504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:11:42.426852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:52257 (local address 127.0.98.1:41809)
0924 21:11:42.426921 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0924 21:11:42.426923 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 21:11:42.426934 (+ 11us) server_negotiation.cc:407] Connection header received
0924 21:11:42.426963 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:11:42.426964 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:11:42.426992 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:11:42.427055 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 21:11:42.427220 (+ 165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:11:42.427781 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:11:42.428685 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:11:42.429117 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:11:42.429220 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:11:42.429230 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0924 21:11:42.429403 (+ 173us) server_negotiation.cc:299] Negotiation successful
0924 21:11:42.429443 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260924 21:11:42.429539 20516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:11:42.426784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36341 (local address 127.0.98.2:48547)
0924 21:11:42.426962 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0924 21:11:42.426971 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 21:11:42.427045 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 21:11:42.427204 (+ 159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 21:11:42.427205 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 21:11:42.427206 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 21:11:42.427275 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:11:42.427276 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:11:42.427864 (+ 588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:11:42.427865 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:11:42.428559 (+ 694us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:11:42.428561 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:11:42.429192 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:11:42.429193 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:11:42.429275 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:11:42.429280 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 21:11:42.429335 (+ 55us) client_negotiation.cc:240] Negotiation successful
0924 21:11:42.429408 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":165,"mutex_wait_us":1,"thread_start_us":115,"threads_started":1}
I20260924 21:11:42.429898 20185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f232d90ec14249e29715dd97b936321b" candidate_uuid: "80fb8f02fad44a87a650e8a715951ab4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce2fa930bc8b4badbfda1890515be8fb" is_pre_election: true
I20260924 21:11:42.430049 20185 raft_consensus.cc:2454] T f232d90ec14249e29715dd97b936321b P ce2fa930bc8b4badbfda1890515be8fb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 80fb8f02fad44a87a650e8a715951ab4 in term 0.
I20260924 21:11:42.430159 20447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f232d90ec14249e29715dd97b936321b" candidate_uuid: "80fb8f02fad44a87a650e8a715951ab4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d5d3b999a0d40a8a3e514f41fd51cdf" is_pre_election: true
I20260924 21:11:42.430279 20252 leader_election.cc:302] T f232d90ec14249e29715dd97b936321b P 80fb8f02fad44a87a650e8a715951ab4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 80fb8f02fad44a87a650e8a715951ab4, ce2fa930bc8b4badbfda1890515be8fb; no voters:
I20260924 21:11:42.430323 20447 raft_consensus.cc:2454] T f232d90ec14249e29715dd97b936321b P 2d5d3b999a0d40a8a3e514f41fd51cdf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 80fb8f02fad44a87a650e8a715951ab4 in term 0.
I20260924 21:11:42.430395 20514 raft_consensus.cc:2781] T f232d90ec14249e29715dd97b936321b P 80fb8f02fad44a87a650e8a715951ab4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 21:11:42.430451 20514 raft_consensus.cc:482] T f232d90ec14249e29715dd97b936321b P 80fb8f02fad44a87a650e8a715951ab4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 21:11:42.430474 20514 raft_consensus.cc:3037] T f232d90ec14249e29715dd97b936321b P 80fb8f02fad44a87a650e8a715951ab4 [term 0 FOLLOWER]: Advancing to term 1
I20260924 21:11:42.431114 20514 raft_consensus.cc:504] T f232d90ec14249e29715dd97b936321b P 80fb8f02fad44a87a650e8a715951ab4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fb8f02fad44a87a650e8a715951ab4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41073 } } peers { permanent_uuid: "ce2fa930bc8b4badbfda1890515be8fb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41809 } } peers { permanent_uuid: "2d5d3b999a0d40a8a3e514f41fd51cdf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36341 } }
I20260924 21:11:42.431242 20514 leader_election.cc:288] T f232d90ec14249e29715dd97b936321b P 80fb8f02fad44a87a650e8a715951ab4 [CANDIDATE]: Term 1 election: Requested vote from peers ce2fa930bc8b4badbfda1890515be8fb (127.0.98.1:41809), 2d5d3b999a0d40a8a3e514f41fd51cdf (127.0.98.3:36341)
I20260924 21:11:42.431406 20447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f232d90ec14249e29715dd97b936321b" candidate_uuid: "80fb8f02fad44a87a650e8a715951ab4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d5d3b999a0d40a8a3e514f41fd51cdf"
I20260924 21:11:42.431406 20185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f232d90ec14249e29715dd97b936321b" candidate_uuid: "80fb8f02fad44a87a650e8a715951ab4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce2fa930bc8b4badbfda1890515be8fb"
I20260924 21:11:42.431486 20447 raft_consensus.cc:3037] T f232d90ec14249e29715dd97b936321b P 2d5d3b999a0d40a8a3e514f41fd51cdf [term 0 FOLLOWER]: Advancing to term 1
I20260924 21:11:42.431486 20185 raft_consensus.cc:3037] T f232d90ec14249e29715dd97b936321b P ce2fa930bc8b4badbfda1890515be8fb [term 0 FOLLOWER]: Advancing to term 1
I20260924 21:11:42.432333 20447 raft_consensus.cc:2454] T f232d90ec14249e29715dd97b936321b P 2d5d3b999a0d40a8a3e514f41fd51cdf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 80fb8f02fad44a87a650e8a715951ab4 in term 1.
I20260924 21:11:42.432333 20185 raft_consensus.cc:2454] T f232d90ec14249e29715dd97b936321b P ce2fa930bc8b4badbfda1890515be8fb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 80fb8f02fad44a87a650e8a715951ab4 in term 1.
I20260924 21:11:42.432495 20250 leader_election.cc:302] T f232d90ec14249e29715dd97b936321b P 80fb8f02fad44a87a650e8a715951ab4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2d5d3b999a0d40a8a3e514f41fd51cdf, 80fb8f02fad44a87a650e8a715951ab4; no voters:
I20260924 21:11:42.432662 20514 raft_consensus.cc:2781] T f232d90ec14249e29715dd97b936321b P 80fb8f02fad44a87a650e8a715951ab4 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 21:11:42.432823 20514 raft_consensus.cc:686] T f232d90ec14249e29715dd97b936321b P 80fb8f02fad44a87a650e8a715951ab4 [term 1 LEADER]: Becoming Leader. State: Replica: 80fb8f02fad44a87a650e8a715951ab4, State: Running, Role: LEADER
I20260924 21:11:42.432996 20514 consensus_queue.cc:237] T f232d90ec14249e29715dd97b936321b P 80fb8f02fad44a87a650e8a715951ab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fb8f02fad44a87a650e8a715951ab4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41073 } } peers { permanent_uuid: "ce2fa930bc8b4badbfda1890515be8fb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41809 } } peers { permanent_uuid: "2d5d3b999a0d40a8a3e514f41fd51cdf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36341 } }
I20260924 21:11:42.433753 20043 catalog_manager.cc:5495] T f232d90ec14249e29715dd97b936321b P 80fb8f02fad44a87a650e8a715951ab4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 80fb8f02fad44a87a650e8a715951ab4 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "80fb8f02fad44a87a650e8a715951ab4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fb8f02fad44a87a650e8a715951ab4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41073 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ce2fa930bc8b4badbfda1890515be8fb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41809 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2d5d3b999a0d40a8a3e514f41fd51cdf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36341 } health_report { overall_health: UNKNOWN } } }
I20260924 21:11:42.453474 20503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:11:42.451386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52788 (local address 127.0.98.2:41073)
0924 21:11:42.451422 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0924 21:11:42.451424 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 21:11:42.451431 (+ 7us) server_negotiation.cc:407] Connection header received
0924 21:11:42.451477 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:11:42.451480 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:11:42.451496 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:11:42.451568 (+ 72us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 21:11:42.451681 (+ 113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:11:42.452088 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:11:42.452777 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:11:42.452944 (+ 167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:11:42.453021 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:11:42.453174 (+ 153us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 21:11:42.453251 (+ 77us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 21:11:42.453329 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0924 21:11:42.453345 (+ 16us) server_negotiation.cc:299] Negotiation successful
0924 21:11:42.453378 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260924 21:11:42.455026 20447 raft_consensus.cc:1261] T f232d90ec14249e29715dd97b936321b P 2d5d3b999a0d40a8a3e514f41fd51cdf [term 1 FOLLOWER]: Refusing update from remote peer 80fb8f02fad44a87a650e8a715951ab4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 21:11:42.455062 20185 raft_consensus.cc:1261] T f232d90ec14249e29715dd97b936321b P ce2fa930bc8b4badbfda1890515be8fb [term 1 FOLLOWER]: Refusing update from remote peer 80fb8f02fad44a87a650e8a715951ab4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 21:11:42.455302 20514 consensus_queue.cc:1035] T f232d90ec14249e29715dd97b936321b P 80fb8f02fad44a87a650e8a715951ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d5d3b999a0d40a8a3e514f41fd51cdf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36341 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 21:11:42.455415 20517 consensus_queue.cc:1035] T f232d90ec14249e29715dd97b936321b P 80fb8f02fad44a87a650e8a715951ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce2fa930bc8b4badbfda1890515be8fb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41809 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260924 21:11:42.455703 20363 tablet.cc:2369] T f232d90ec14249e29715dd97b936321b P 80fb8f02fad44a87a650e8a715951ab4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 21:11:42.457891 20522 mvcc.cc:205] Tried to move back new op lower bound from 7333004502853427200 to 7333004502766182400. Current Snapshot: MvccSnapshot[applied={T|T < 7333004502853427200}]
I20260924 21:11:42.459929 20042 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35256:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 21:11:42.460006 20042 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35256:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 21:11:42.460614 20042 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c459f6c1ce8949fb9a392c55178fbdd2: Sending DeleteTablet for 3 replicas of tablet f232d90ec14249e29715dd97b936321b
I20260924 21:11:42.460875 20165 tablet_service.cc:1514] Processing DeleteTablet for tablet f232d90ec14249e29715dd97b936321b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 21:11:42 UTC) from {username='slave'} at 127.0.0.1:51380
I20260924 21:11:42.460882 20296 tablet_service.cc:1514] Processing DeleteTablet for tablet f232d90ec14249e29715dd97b936321b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 21:11:42 UTC) from {username='slave'} at 127.0.0.1:52786
I20260924 21:11:42.460917 20427 tablet_service.cc:1514] Processing DeleteTablet for tablet f232d90ec14249e29715dd97b936321b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 21:11:42 UTC) from {username='slave'} at 127.0.0.1:39242
I20260924 21:11:42.461086 20531 tablet_replica.cc:330] T f232d90ec14249e29715dd97b936321b P 2d5d3b999a0d40a8a3e514f41fd51cdf: stopping tablet replica
I20260924 21:11:42.461169 20531 raft_consensus.cc:2229] T f232d90ec14249e29715dd97b936321b P 2d5d3b999a0d40a8a3e514f41fd51cdf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 21:11:42.461249 20531 raft_consensus.cc:2258] T f232d90ec14249e29715dd97b936321b P 2d5d3b999a0d40a8a3e514f41fd51cdf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 21:11:42.461290 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAzNNjS/build/release/bin/kudu with pid 20103
I20260924 21:11:42.461776 20532 tablet_replica.cc:330] T f232d90ec14249e29715dd97b936321b P 80fb8f02fad44a87a650e8a715951ab4: stopping tablet replica
I20260924 21:11:42.461877 20532 raft_consensus.cc:2229] T f232d90ec14249e29715dd97b936321b P 80fb8f02fad44a87a650e8a715951ab4 [term 1 LEADER]: Raft consensus shutting down.
I20260924 21:11:42.462126 20531 ts_tablet_manager.cc:1905] T f232d90ec14249e29715dd97b936321b P 2d5d3b999a0d40a8a3e514f41fd51cdf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 21:11:42.463900 20531 ts_tablet_manager.cc:1918] T f232d90ec14249e29715dd97b936321b P 2d5d3b999a0d40a8a3e514f41fd51cdf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 21:11:42.463969 20531 log.cc:1196] T f232d90ec14249e29715dd97b936321b P 2d5d3b999a0d40a8a3e514f41fd51cdf: Deleting WAL directory at /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790284076800039-392-0/minicluster-data/ts-2/wal/wals/f232d90ec14249e29715dd97b936321b
I20260924 21:11:42.464198 20531 ts_tablet_manager.cc:1939] T f232d90ec14249e29715dd97b936321b P 2d5d3b999a0d40a8a3e514f41fd51cdf: Deleting consensus metadata
I20260924 21:11:42.464389 20532 raft_consensus.cc:2258] T f232d90ec14249e29715dd97b936321b P 80fb8f02fad44a87a650e8a715951ab4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 21:11:42.464744 20532 ts_tablet_manager.cc:1905] T f232d90ec14249e29715dd97b936321b P 80fb8f02fad44a87a650e8a715951ab4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 21:11:42.465040 20028 catalog_manager.cc:4841] TS 2d5d3b999a0d40a8a3e514f41fd51cdf (127.0.98.3:36341): tablet f232d90ec14249e29715dd97b936321b (table test-table [id=c9a56f232f9a47bfaafbaa878bf78889]) successfully deleted
I20260924 21:11:42.466573 20532 ts_tablet_manager.cc:1918] T f232d90ec14249e29715dd97b936321b P 80fb8f02fad44a87a650e8a715951ab4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 21:11:42.466640 20532 log.cc:1196] T f232d90ec14249e29715dd97b936321b P 80fb8f02fad44a87a650e8a715951ab4: Deleting WAL directory at /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790284076800039-392-0/minicluster-data/ts-1/wal/wals/f232d90ec14249e29715dd97b936321b
I20260924 21:11:42.466877 20532 ts_tablet_manager.cc:1939] T f232d90ec14249e29715dd97b936321b P 80fb8f02fad44a87a650e8a715951ab4: Deleting consensus metadata
I20260924 21:11:42.467366 20028 catalog_manager.cc:4841] TS 80fb8f02fad44a87a650e8a715951ab4 (127.0.98.2:41073): tablet f232d90ec14249e29715dd97b936321b (table test-table [id=c9a56f232f9a47bfaafbaa878bf78889]) successfully deleted
W20260924 21:11:42.468788 20030 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:41809 (error 108)
I20260924 21:11:42.469249 20505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:11:42.469119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41809 (local address 127.0.0.1:51392)
0924 21:11:42.469175 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0924 21:11:42.469217 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:41809: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260924 21:11:42.469316 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAzNNjS/build/release/bin/kudu with pid 20234
W20260924 21:11:42.471905 20030 catalog_manager.cc:4584] TS ce2fa930bc8b4badbfda1890515be8fb (127.0.98.1:41809): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet f232d90ec14249e29715dd97b936321b: Network error: Client connection negotiation failed: client connection to 127.0.98.1:41809: connect: Connection refused (error 111)
I20260924 21:11:42.475001 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAzNNjS/build/release/bin/kudu with pid 20365
I20260924 21:11:42.479121 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAzNNjS/build/release/bin/kudu with pid 20012
2026-09-24T21:11:42Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (650 ms)
[----------] 6 tests from EncryptionPolicyTest (7601 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790284076800039-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 21:11:42 dist-test-slave-c7tk krb5kdc[20536](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 21:11:42 dist-test-slave-c7tk krb5kdc[20536](info): set up 2 sockets
Sep 24 21:11:42 dist-test-slave-c7tk krb5kdc[20536](info): commencing operation
krb5kdc: starting...
W20260924 21:11:43.397158 392 mini_kdc.cc:121] Time spent starting KDC: real 0.900s user 0.000s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790284076800039-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790284076800039-392-0/krb5kdc/test-user.keytab.
Sep 24 21:11:43 dist-test-slave-c7tk krb5kdc[20536](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284303, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T21:11:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T21:11:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790284076800039-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790284076800039-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 21:11:43.498746 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAzNNjS/build/release/bin/kudu
/tmp/dist-test-taskAzNNjS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790284076800039-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790284076800039-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790284076800039-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790284076800039-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37535
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790284076800039-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44193
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37535
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790284076800039-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790284076800039-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790284076800039-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 21:11:43.573076 20549 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 21:11:43.573273 20549 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 21:11:43.573307 20549 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 21:11:43.574349 20549 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 21:11:43.574409 20549 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 21:11:43.574430 20549 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 21:11:43.574455 20549 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 21:11:43.574483 20549 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 21:11:43.575757 20549 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44193
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790284076800039-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790284076800039-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37535
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790284076800039-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37535
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790284076800039-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20549
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790284076800039-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 19:49:06 UTC on ccycloud.quasar-vhqjoc.root.comops.site
build id 4670864
I20260924 21:11:43.575973 20549 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 21:11:43.576184 20549 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 21:11:43.580787 20557 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 21:11:43.580801 20555 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 21:11:43.580744 20554 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 21:11:43.580876 20549 server_base.cc:1034] running on GCE node
I20260924 21:11:43.581187 20549 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 21:11:43.581489 20549 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 21:11:43.582628 20549 hybrid_clock.cc:648] HybridClock initialized: now 1790284303582613 us; error 39 us; skew 500 ppm
Sep 24 21:11:43 dist-test-slave-c7tk krb5kdc[20536](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284303, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 21:11:43.588467 20549 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 21:11:43.588927 20549 webserver.cc:460] Webserver started at http://127.0.0.1:32961/ using document root <none> and password file <none>
I20260924 21:11:43.589148 20549 fs_manager.cc:359] Metadata directory not provided
I20260924 21:11:43.589214 20549 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 21:11:43.589357 20549 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 21:11:43.590389 20549 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790284076800039-392-0/minicluster-data/master-0/data/instance:
uuid: "ab998588ba1449d0a5da2ec98ce7e615"
format_stamp: "Formatted at 2026-09-24 21:11:43 on dist-test-slave-c7tk"
server_key: "01b5d3968969cecafb4197daf99d644d"
server_key_iv: "4d012582a47d8fc51cc151fee7a99442"
server_key_version: "encryptionkey@0"
I20260924 21:11:43.590724 20549 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790284076800039-392-0/minicluster-data/master-0/wal/instance:
uuid: "ab998588ba1449d0a5da2ec98ce7e615"
format_stamp: "Formatted at 2026-09-24 21:11:43 on dist-test-slave-c7tk"
server_key: "01b5d3968969cecafb4197daf99d644d"
server_key_iv: "4d012582a47d8fc51cc151fee7a99442"
server_key_version: "encryptionkey@0"
I20260924 21:11:43.592000 20549 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 21:11:43.592671 20564 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 21:11:43.592854 20549 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 21:11:43.592941 20549 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790284076800039-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790284076800039-392-0/minicluster-data/master-0/wal
uuid: "ab998588ba1449d0a5da2ec98ce7e615"
format_stamp: "Formatted at 2026-09-24 21:11:43 on dist-test-slave-c7tk"
server_key: "01b5d3968969cecafb4197daf99d644d"
server_key_iv: "4d012582a47d8fc51cc151fee7a99442"
server_key_version: "encryptionkey@0"
I20260924 21:11:43.593008 20549 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790284076800039-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790284076800039-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790284076800039-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 21:11:43.606261 20549 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 21:11:43.607669 20549 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 21:11:43.607806 20549 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 21:11:43.611657 20549 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37535
I20260924 21:11:43.611683 20616 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37535 every 8 connection(s)
I20260924 21:11:43.611982 20549 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790284076800039-392-0/minicluster-data/master-0/data/info.pb
I20260924 21:11:43.612437 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAzNNjS/build/release/bin/kudu as pid 20549
I20260924 21:11:43.612560 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790284076800039-392-0/minicluster-data/master-0/wal/instance
I20260924 21:11:43.612747 392 external_mini_cluster.cc:1428] Setting key 2b9ff9bca343e4e0d16bbdf0d3b74e67
I20260924 21:11:43.612968 20617 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 21:11:43.615623 20617 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ab998588ba1449d0a5da2ec98ce7e615: Bootstrap starting.
I20260924 21:11:43.616155 20617 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ab998588ba1449d0a5da2ec98ce7e615: Neither blocks nor log segments found. Creating new log.
I20260924 21:11:43.616451 20617 log.cc:824] T 00000000000000000000000000000000 P ab998588ba1449d0a5da2ec98ce7e615: Log is configured to *not* fsync() on all Append() calls
I20260924 21:11:43.617040 20617 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ab998588ba1449d0a5da2ec98ce7e615: No bootstrap required, opened a new log
Sep 24 21:11:43 dist-test-slave-c7tk krb5kdc[20536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284303, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260924 21:11:43.618891 20617 raft_consensus.cc:348] T 00000000000000000000000000000000 P ab998588ba1449d0a5da2ec98ce7e615 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab998588ba1449d0a5da2ec98ce7e615" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37535 } }
I20260924 21:11:43.619033 20617 raft_consensus.cc:374] T 00000000000000000000000000000000 P ab998588ba1449d0a5da2ec98ce7e615 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 21:11:43.619078 20617 raft_consensus.cc:729] T 00000000000000000000000000000000 P ab998588ba1449d0a5da2ec98ce7e615 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab998588ba1449d0a5da2ec98ce7e615, State: Initialized, Role: FOLLOWER
I20260924 21:11:43.619201 20617 consensus_queue.cc:260] T 00000000000000000000000000000000 P ab998588ba1449d0a5da2ec98ce7e615 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab998588ba1449d0a5da2ec98ce7e615" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37535 } }
I20260924 21:11:43.619297 20617 raft_consensus.cc:388] T 00000000000000000000000000000000 P ab998588ba1449d0a5da2ec98ce7e615 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 21:11:43.619333 20617 raft_consensus.cc:482] T 00000000000000000000000000000000 P ab998588ba1449d0a5da2ec98ce7e615 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 21:11:43.619385 20617 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ab998588ba1449d0a5da2ec98ce7e615 [term 0 FOLLOWER]: Advancing to term 1
I20260924 21:11:43.620113 20617 raft_consensus.cc:504] T 00000000000000000000000000000000 P ab998588ba1449d0a5da2ec98ce7e615 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab998588ba1449d0a5da2ec98ce7e615" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37535 } }
I20260924 21:11:43.620220 20617 leader_election.cc:302] T 00000000000000000000000000000000 P ab998588ba1449d0a5da2ec98ce7e615 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ab998588ba1449d0a5da2ec98ce7e615; no voters:
I20260924 21:11:43.620359 20617 leader_election.cc:288] T 00000000000000000000000000000000 P ab998588ba1449d0a5da2ec98ce7e615 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 21:11:43.620431 20622 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ab998588ba1449d0a5da2ec98ce7e615 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 21:11:43.620566 20622 raft_consensus.cc:686] T 00000000000000000000000000000000 P ab998588ba1449d0a5da2ec98ce7e615 [term 1 LEADER]: Becoming Leader. State: Replica: ab998588ba1449d0a5da2ec98ce7e615, State: Running, Role: LEADER
I20260924 21:11:43.620579 20617 sys_catalog.cc:564] T 00000000000000000000000000000000 P ab998588ba1449d0a5da2ec98ce7e615 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 21:11:43.620644 20622 consensus_queue.cc:237] T 00000000000000000000000000000000 P ab998588ba1449d0a5da2ec98ce7e615 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab998588ba1449d0a5da2ec98ce7e615" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37535 } }
I20260924 21:11:43.620966 20623 sys_catalog.cc:455] T 00000000000000000000000000000000 P ab998588ba1449d0a5da2ec98ce7e615 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ab998588ba1449d0a5da2ec98ce7e615" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab998588ba1449d0a5da2ec98ce7e615" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37535 } } }
I20260924 21:11:43.621064 20623 sys_catalog.cc:458] T 00000000000000000000000000000000 P ab998588ba1449d0a5da2ec98ce7e615 [sys.catalog]: This master's current role is: LEADER
I20260924 21:11:43.621289 20624 sys_catalog.cc:455] T 00000000000000000000000000000000 P ab998588ba1449d0a5da2ec98ce7e615 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ab998588ba1449d0a5da2ec98ce7e615. Latest consensus state: current_term: 1 leader_uuid: "ab998588ba1449d0a5da2ec98ce7e615" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab998588ba1449d0a5da2ec98ce7e615" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37535 } } }
I20260924 21:11:43.621423 20624 sys_catalog.cc:458] T 00000000000000000000000000000000 P ab998588ba1449d0a5da2ec98ce7e615 [sys.catalog]: This master's current role is: LEADER
I20260924 21:11:43.621567 20629 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 21:11:43.622040 20629 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 21:11:43.622998 20620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:11:43.613384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43495 (local address 127.0.0.1:37535)
0924 21:11:43.613563 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0924 21:11:43.613572 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 21:11:43.613604 (+ 32us) server_negotiation.cc:407] Connection header received
0924 21:11:43.614153 (+ 549us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:11:43.614162 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:11:43.614214 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:11:43.614353 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:11:43.614921 (+ 568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:11:43.615776 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:11:43.616539 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:11:43.617008 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:11:43.617129 (+ 121us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:11:43.619192 (+ 2063us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 21:11:43.619195 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 21:11:43.619195 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 21:11:43.619206 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 21:11:43.620953 (+ 1747us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:11:43.621396 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:11:43.621399 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:11:43.621399 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:11:43.621495 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:11:43.621787 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:11:43.621788 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:11:43.621788 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:11:43.622139 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 21:11:43.622304 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0924 21:11:43.622695 (+ 391us) server_negotiation.cc:299] Negotiation successful
0924 21:11:43.622784 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":83,"threads_started":1}
I20260924 21:11:43.623901 20629 catalog_manager.cc:1347] Generated new cluster ID: 6e23f1b1daf246f8b4e8063e7ba9f94e
I20260924 21:11:43.623955 20629 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 21:11:43.623946 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 21:11:43.635557 20629 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 21:11:43.636044 20629 catalog_manager.cc:1504] Loading token signing keys...
I20260924 21:11:43.639544 20629 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ab998588ba1449d0a5da2ec98ce7e615: Generated new TSK 0
I20260924 21:11:43.639715 20629 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 21:11:43.641999 20620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:11:43.626102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39671 (local address 127.0.0.1:37535)
0924 21:11:43.626135 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0924 21:11:43.626137 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 21:11:43.635280 (+ 9143us) server_negotiation.cc:407] Connection header received
0924 21:11:43.635491 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:11:43.635492 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:11:43.635501 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:11:43.635549 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:11:43.636277 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:11:43.636938 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:11:43.637947 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:11:43.638263 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:11:43.638329 (+ 66us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:11:43.639743 (+ 1414us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 21:11:43.639745 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 21:11:43.639745 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 21:11:43.639752 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 21:11:43.640981 (+ 1229us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:11:43.641345 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:11:43.641346 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:11:43.641346 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:11:43.641389 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:11:43.641583 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:11:43.641583 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:11:43.641583 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:11:43.641661 (+ 78us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 21:11:43.641727 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0924 21:11:43.641882 (+ 155us) server_negotiation.cc:299] Negotiation successful
0924 21:11:43.641933 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260924 21:11:43.643673 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAzNNjS/build/release/bin/kudu with pid 20549
W20260924 21:11:43.647426 20534 connection.cc:535] client connection to 127.0.0.1:37535 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:37535): Connection reset by peer (error 104)
2026-09-24T21:11:43Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1163 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790284076800039-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 21:11:43 dist-test-slave-c7tk krb5kdc[20648](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 21:11:43 dist-test-slave-c7tk krb5kdc[20648](info): set up 2 sockets
Sep 24 21:11:43 dist-test-slave-c7tk krb5kdc[20648](info): commencing operation
krb5kdc: starting...
W20260924 21:11:44.589341 392 mini_kdc.cc:121] Time spent starting KDC: real 0.928s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790284076800039-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790284076800039-392-0/krb5kdc/test-user.keytab.
Sep 24 21:11:44 dist-test-slave-c7tk krb5kdc[20648](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284304, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T21:11:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T21:11:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790284076800039-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790284076800039-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 21:11:44.690095 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAzNNjS/build/release/bin/kudu
/tmp/dist-test-taskAzNNjS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790284076800039-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790284076800039-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790284076800039-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790284076800039-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37483
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790284076800039-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39619
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37483
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790284076800039-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790284076800039-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790284076800039-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 21:11:44.761185 20661 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 21:11:44.761368 20661 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 21:11:44.761399 20661 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 21:11:44.762373 20661 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 21:11:44.762429 20661 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 21:11:44.762450 20661 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 21:11:44.762470 20661 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 21:11:44.762502 20661 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 21:11:44.763702 20661 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39619
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790284076800039-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790284076800039-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37483
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790284076800039-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37483
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790284076800039-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20661
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790284076800039-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 19:49:06 UTC on ccycloud.quasar-vhqjoc.root.comops.site
build id 4670864
I20260924 21:11:44.763963 20661 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 21:11:44.764154 20661 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 21:11:44.768220 20661 server_base.cc:1034] running on GCE node
W20260924 21:11:44.768206 20666 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 21:11:44.768193 20667 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 21:11:44.768203 20669 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 21:11:44.768572 20661 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 21:11:44.768886 20661 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 21:11:44.770066 20661 hybrid_clock.cc:648] HybridClock initialized: now 1790284304770052 us; error 42 us; skew 500 ppm
Sep 24 21:11:44 dist-test-slave-c7tk krb5kdc[20648](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284304, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 21:11:44.775477 20661 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 21:11:44.775894 20661 webserver.cc:460] Webserver started at http://127.0.0.1:44635/ using document root <none> and password file <none>
I20260924 21:11:44.776108 20661 fs_manager.cc:359] Metadata directory not provided
I20260924 21:11:44.776168 20661 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 21:11:44.776300 20661 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 21:11:44.777192 20661 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790284076800039-392-0/minicluster-data/master-0/data/instance:
uuid: "d42775ecd5b4401ca3fe62962d49c29c"
format_stamp: "Formatted at 2026-09-24 21:11:44 on dist-test-slave-c7tk"
server_key: "8a8c03ef312590ba2fa71e6357d8bd11"
server_key_iv: "54a525943b763ec7e29b7b461ba08c7d"
server_key_version: "encryptionkey@0"
I20260924 21:11:44.777489 20661 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790284076800039-392-0/minicluster-data/master-0/wal/instance:
uuid: "d42775ecd5b4401ca3fe62962d49c29c"
format_stamp: "Formatted at 2026-09-24 21:11:44 on dist-test-slave-c7tk"
server_key: "8a8c03ef312590ba2fa71e6357d8bd11"
server_key_iv: "54a525943b763ec7e29b7b461ba08c7d"
server_key_version: "encryptionkey@0"
I20260924 21:11:44.778652 20661 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 21:11:44.779373 20676 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 21:11:44.779527 20661 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 21:11:44.779613 20661 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790284076800039-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790284076800039-392-0/minicluster-data/master-0/wal
uuid: "d42775ecd5b4401ca3fe62962d49c29c"
format_stamp: "Formatted at 2026-09-24 21:11:44 on dist-test-slave-c7tk"
server_key: "8a8c03ef312590ba2fa71e6357d8bd11"
server_key_iv: "54a525943b763ec7e29b7b461ba08c7d"
server_key_version: "encryptionkey@0"
I20260924 21:11:44.779690 20661 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790284076800039-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790284076800039-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790284076800039-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 21:11:44.801456 20661 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 21:11:44.803186 20661 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 21:11:44.803319 20661 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 21:11:44.806816 20661 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37483
I20260924 21:11:44.806854 20728 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37483 every 8 connection(s)
I20260924 21:11:44.807176 20661 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790284076800039-392-0/minicluster-data/master-0/data/info.pb
I20260924 21:11:44.808063 20729 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 21:11:44.810178 20729 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d42775ecd5b4401ca3fe62962d49c29c: Bootstrap starting.
I20260924 21:11:44.810619 20729 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d42775ecd5b4401ca3fe62962d49c29c: Neither blocks nor log segments found. Creating new log.
I20260924 21:11:44.810845 20729 log.cc:824] T 00000000000000000000000000000000 P d42775ecd5b4401ca3fe62962d49c29c: Log is configured to *not* fsync() on all Append() calls
I20260924 21:11:44.811426 20729 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d42775ecd5b4401ca3fe62962d49c29c: No bootstrap required, opened a new log
I20260924 21:11:44.812772 20729 raft_consensus.cc:348] T 00000000000000000000000000000000 P d42775ecd5b4401ca3fe62962d49c29c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d42775ecd5b4401ca3fe62962d49c29c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37483 } }
I20260924 21:11:44.812896 20729 raft_consensus.cc:374] T 00000000000000000000000000000000 P d42775ecd5b4401ca3fe62962d49c29c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 21:11:44.812918 20729 raft_consensus.cc:729] T 00000000000000000000000000000000 P d42775ecd5b4401ca3fe62962d49c29c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d42775ecd5b4401ca3fe62962d49c29c, State: Initialized, Role: FOLLOWER
I20260924 21:11:44.813014 20729 consensus_queue.cc:260] T 00000000000000000000000000000000 P d42775ecd5b4401ca3fe62962d49c29c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d42775ecd5b4401ca3fe62962d49c29c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37483 } }
I20260924 21:11:44.813063 20729 raft_consensus.cc:388] T 00000000000000000000000000000000 P d42775ecd5b4401ca3fe62962d49c29c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 21:11:44.813084 20729 raft_consensus.cc:482] T 00000000000000000000000000000000 P d42775ecd5b4401ca3fe62962d49c29c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 21:11:44.813115 20729 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d42775ecd5b4401ca3fe62962d49c29c [term 0 FOLLOWER]: Advancing to term 1
I20260924 21:11:44.813551 20729 raft_consensus.cc:504] T 00000000000000000000000000000000 P d42775ecd5b4401ca3fe62962d49c29c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d42775ecd5b4401ca3fe62962d49c29c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37483 } }
I20260924 21:11:44.813657 20729 leader_election.cc:302] T 00000000000000000000000000000000 P d42775ecd5b4401ca3fe62962d49c29c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d42775ecd5b4401ca3fe62962d49c29c; no voters:
I20260924 21:11:44.813835 20729 leader_election.cc:288] T 00000000000000000000000000000000 P d42775ecd5b4401ca3fe62962d49c29c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 21:11:44.813849 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAzNNjS/build/release/bin/kudu as pid 20661
I20260924 21:11:44.813889 20732 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d42775ecd5b4401ca3fe62962d49c29c [term 1 FOLLOWER]: Leader election won for term 1
I20260924 21:11:44.813952 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790284076800039-392-0/minicluster-data/master-0/wal/instance
I20260924 21:11:44.814046 20732 raft_consensus.cc:686] T 00000000000000000000000000000000 P d42775ecd5b4401ca3fe62962d49c29c [term 1 LEADER]: Becoming Leader. State: Replica: d42775ecd5b4401ca3fe62962d49c29c, State: Running, Role: LEADER
I20260924 21:11:44.814093 20729 sys_catalog.cc:564] T 00000000000000000000000000000000 P d42775ecd5b4401ca3fe62962d49c29c [sys.catalog]: configured and running, proceeding with master startup.
I20260924 21:11:44.814152 392 external_mini_cluster.cc:1428] Setting key a0a629c51b0fba90058d34497df2973b
I20260924 21:11:44.814155 20732 consensus_queue.cc:237] T 00000000000000000000000000000000 P d42775ecd5b4401ca3fe62962d49c29c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d42775ecd5b4401ca3fe62962d49c29c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37483 } }
I20260924 21:11:44.814988 20733 sys_catalog.cc:455] T 00000000000000000000000000000000 P d42775ecd5b4401ca3fe62962d49c29c [sys.catalog]: SysCatalogTable state changed. Reason: New leader d42775ecd5b4401ca3fe62962d49c29c. Latest consensus state: current_term: 1 leader_uuid: "d42775ecd5b4401ca3fe62962d49c29c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d42775ecd5b4401ca3fe62962d49c29c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37483 } } }
I20260924 21:11:44.815088 20733 sys_catalog.cc:458] T 00000000000000000000000000000000 P d42775ecd5b4401ca3fe62962d49c29c [sys.catalog]: This master's current role is: LEADER
I20260924 21:11:44.815407 20734 sys_catalog.cc:455] T 00000000000000000000000000000000 P d42775ecd5b4401ca3fe62962d49c29c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d42775ecd5b4401ca3fe62962d49c29c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d42775ecd5b4401ca3fe62962d49c29c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37483 } } }
I20260924 21:11:44.815500 20742 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 21:11:44.815507 20734 sys_catalog.cc:458] T 00000000000000000000000000000000 P d42775ecd5b4401ca3fe62962d49c29c [sys.catalog]: This master's current role is: LEADER
I20260924 21:11:44.815989 20742 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 21:11:44.817257 20742 catalog_manager.cc:1347] Generated new cluster ID: 32ef9b4601364ae89cf9afc19f1c8c66
I20260924 21:11:44.817301 20742 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 21:11:44.819793 20736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:11:44.814584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48279 (local address 127.0.0.1:37483)
0924 21:11:44.814962 (+ 378us) server_negotiation.cc:206] Beginning negotiation
0924 21:11:44.814970 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 21:11:44.814999 (+ 29us) server_negotiation.cc:407] Connection header received
0924 21:11:44.815198 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:11:44.815204 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:11:44.815257 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:11:44.815384 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:11:44.815829 (+ 445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:11:44.816924 (+ 1095us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:11:44.817892 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:11:44.818383 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:11:44.818468 (+ 85us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:11:44.819144 (+ 676us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 21:11:44.819148 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 21:11:44.819149 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 21:11:44.819339 (+ 190us) server_negotiation.cc:922] calling sasl_server_start()
0924 21:11:44.819372 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 21:11:44.819384 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 21:11:44.819464 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0924 21:11:44.819587 (+ 123us) server_negotiation.cc:299] Negotiation successful
0924 21:11:44.819611 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":336,"thread_start_us":60,"threads_started":1}
I20260924 21:11:44.825507 20742 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 21:11:44.826102 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 21:11:44.826165 20742 catalog_manager.cc:1504] Loading token signing keys...
I20260924 21:11:44.830713 20736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:11:44.827494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35993 (local address 127.0.0.1:37483)
0924 21:11:44.827544 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0924 21:11:44.827545 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 21:11:44.827640 (+ 95us) server_negotiation.cc:407] Connection header received
0924 21:11:44.827667 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:11:44.827667 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:11:44.827677 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:11:44.827744 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:11:44.827841 (+ 97us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:11:44.828479 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:11:44.829474 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:11:44.829782 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:11:44.829877 (+ 95us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:11:44.830483 (+ 606us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 21:11:44.830484 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 21:11:44.830485 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 21:11:44.830489 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 21:11:44.830502 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 21:11:44.830506 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 21:11:44.830572 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0924 21:11:44.830636 (+ 64us) server_negotiation.cc:299] Negotiation successful
0924 21:11:44.830647 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260924 21:11:44.831441 20742 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d42775ecd5b4401ca3fe62962d49c29c: Generated new TSK 0
I20260924 21:11:44.831575 20742 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 21:11:44.837756 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAzNNjS/build/release/bin/kudu with pid 20661
W20260924 21:11:44.841572 20646 connection.cc:535] client connection to 127.0.0.1:37483 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:37483): Connection reset by peer (error 104)
2026-09-24T21:11:44Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1194 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790284076800039-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 21:11:44 dist-test-slave-c7tk krb5kdc[20760](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 21:11:44 dist-test-slave-c7tk krb5kdc[20760](info): set up 2 sockets
Sep 24 21:11:44 dist-test-slave-c7tk krb5kdc[20760](info): commencing operation
krb5kdc: starting...
W20260924 21:11:45.765184 392 mini_kdc.cc:121] Time spent starting KDC: real 0.910s user 0.004s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790284076800039-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790284076800039-392-0/krb5kdc/test-user.keytab.
Sep 24 21:11:45 dist-test-slave-c7tk krb5kdc[20760](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284305, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T21:11:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T21:11:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790284076800039-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790284076800039-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 21:11:45.866593 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAzNNjS/build/release/bin/kudu
/tmp/dist-test-taskAzNNjS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790284076800039-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790284076800039-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790284076800039-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790284076800039-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34841
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790284076800039-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40381
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34841
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790284076800039-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790284076800039-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790284076800039-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 21:11:45.936378 20773 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 21:11:45.936527 20773 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 21:11:45.936545 20773 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 21:11:45.937500 20773 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 21:11:45.937536 20773 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 21:11:45.937547 20773 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 21:11:45.937558 20773 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 21:11:45.937603 20773 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 21:11:45.938802 20773 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40381
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790284076800039-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790284076800039-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34841
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790284076800039-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34841
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790284076800039-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20773
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790284076800039-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 19:49:06 UTC on ccycloud.quasar-vhqjoc.root.comops.site
build id 4670864
I20260924 21:11:45.939064 20773 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 21:11:45.939270 20773 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 21:11:45.943308 20778 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 21:11:45.943445 20773 server_base.cc:1034] running on GCE node
W20260924 21:11:45.943341 20781 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 21:11:45.943308 20779 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 21:11:45.943725 20773 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 21:11:45.944007 20773 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 21:11:45.945161 20773 hybrid_clock.cc:648] HybridClock initialized: now 1790284305945152 us; error 32 us; skew 500 ppm
Sep 24 21:11:45 dist-test-slave-c7tk krb5kdc[20760](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284305, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 21:11:45.947190 20773 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 21:11:45.947549 20773 webserver.cc:460] Webserver started at http://127.0.0.1:34741/ using document root <none> and password file <none>
I20260924 21:11:45.947752 20773 fs_manager.cc:359] Metadata directory not provided
I20260924 21:11:45.947809 20773 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 21:11:45.947948 20773 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 21:11:45.948760 20773 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790284076800039-392-0/minicluster-data/master-0/data/instance:
uuid: "a0ce0917937449c6af20a4e4d089973e"
format_stamp: "Formatted at 2026-09-24 21:11:45 on dist-test-slave-c7tk"
server_key: "5ad8a58da102e790ab1b87137c0c18aa"
server_key_iv: "331732ebba488d71f5a67c2be3515f41"
server_key_version: "encryptionkey@0"
I20260924 21:11:45.949108 20773 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790284076800039-392-0/minicluster-data/master-0/wal/instance:
uuid: "a0ce0917937449c6af20a4e4d089973e"
format_stamp: "Formatted at 2026-09-24 21:11:45 on dist-test-slave-c7tk"
server_key: "5ad8a58da102e790ab1b87137c0c18aa"
server_key_iv: "331732ebba488d71f5a67c2be3515f41"
server_key_version: "encryptionkey@0"
I20260924 21:11:45.950408 20773 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 21:11:45.950984 20788 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 21:11:45.951131 20773 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 21:11:45.951211 20773 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790284076800039-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790284076800039-392-0/minicluster-data/master-0/wal
uuid: "a0ce0917937449c6af20a4e4d089973e"
format_stamp: "Formatted at 2026-09-24 21:11:45 on dist-test-slave-c7tk"
server_key: "5ad8a58da102e790ab1b87137c0c18aa"
server_key_iv: "331732ebba488d71f5a67c2be3515f41"
server_key_version: "encryptionkey@0"
I20260924 21:11:45.951278 20773 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790284076800039-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790284076800039-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790284076800039-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 21:11:45.954069 20773 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 21:11:45.955740 20773 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 21:11:45.955873 20773 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 21:11:45.959301 20773 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34841
I20260924 21:11:45.959328 20840 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34841 every 8 connection(s)
I20260924 21:11:45.959615 20773 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790284076800039-392-0/minicluster-data/master-0/data/info.pb
I20260924 21:11:45.959932 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAzNNjS/build/release/bin/kudu as pid 20773
I20260924 21:11:45.960068 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790284076800039-392-0/minicluster-data/master-0/wal/instance
I20260924 21:11:45.960264 392 external_mini_cluster.cc:1428] Setting key 70f28fa78b28cdba8131ad3956263280
I20260924 21:11:45.960665 20841 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 21:11:45.962291 20844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:11:45.960656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57967 (local address 127.0.0.1:34841)
0924 21:11:45.960814 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0924 21:11:45.960820 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 21:11:45.960843 (+ 23us) server_negotiation.cc:407] Connection header received
0924 21:11:45.961198 (+ 355us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:11:45.961203 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:11:45.961223 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:11:45.961351 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:11:45.961902 (+ 551us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 21:11:45.961906 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 21:11:45.961906 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 21:11:45.961910 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 21:11:45.961944 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 21:11:45.961960 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 21:11:45.962040 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0924 21:11:45.962102 (+ 62us) server_negotiation.cc:299] Negotiation successful
0924 21:11:45.962135 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114,"thread_start_us":67,"threads_started":1}
I20260924 21:11:45.963163 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 21:11:45.963356 20841 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a0ce0917937449c6af20a4e4d089973e: Bootstrap starting.
I20260924 21:11:45.963846 20841 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a0ce0917937449c6af20a4e4d089973e: Neither blocks nor log segments found. Creating new log.
I20260924 21:11:45.964107 20841 log.cc:824] T 00000000000000000000000000000000 P a0ce0917937449c6af20a4e4d089973e: Log is configured to *not* fsync() on all Append() calls
I20260924 21:11:45.964777 20841 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a0ce0917937449c6af20a4e4d089973e: No bootstrap required, opened a new log
I20260924 21:11:45.965322 20844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:11:45.964330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56169 (local address 127.0.0.1:34841)
0924 21:11:45.964359 (+ 29us) server_negotiation.cc:206] Beginning negotiation
0924 21:11:45.964361 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 21:11:45.964468 (+ 107us) server_negotiation.cc:407] Connection header received
0924 21:11:45.964559 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:11:45.964560 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:11:45.964564 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:11:45.964632 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:11:45.965146 (+ 514us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 21:11:45.965147 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 21:11:45.965147 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 21:11:45.965151 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 21:11:45.965161 (+ 10us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 21:11:45.965165 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 21:11:45.965208 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0924 21:11:45.965255 (+ 47us) server_negotiation.cc:299] Negotiation successful
0924 21:11:45.965266 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13}
I20260924 21:11:45.966542 20841 raft_consensus.cc:348] T 00000000000000000000000000000000 P a0ce0917937449c6af20a4e4d089973e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0ce0917937449c6af20a4e4d089973e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34841 } }
I20260924 21:11:45.966643 20841 raft_consensus.cc:374] T 00000000000000000000000000000000 P a0ce0917937449c6af20a4e4d089973e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 21:11:45.966662 20841 raft_consensus.cc:729] T 00000000000000000000000000000000 P a0ce0917937449c6af20a4e4d089973e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0ce0917937449c6af20a4e4d089973e, State: Initialized, Role: FOLLOWER
I20260924 21:11:45.966753 20841 consensus_queue.cc:260] T 00000000000000000000000000000000 P a0ce0917937449c6af20a4e4d089973e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0ce0917937449c6af20a4e4d089973e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34841 } }
I20260924 21:11:45.966825 20841 raft_consensus.cc:388] T 00000000000000000000000000000000 P a0ce0917937449c6af20a4e4d089973e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 21:11:45.966856 20841 raft_consensus.cc:482] T 00000000000000000000000000000000 P a0ce0917937449c6af20a4e4d089973e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 21:11:45.966905 20841 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a0ce0917937449c6af20a4e4d089973e [term 0 FOLLOWER]: Advancing to term 1
I20260924 21:11:45.967355 20841 raft_consensus.cc:504] T 00000000000000000000000000000000 P a0ce0917937449c6af20a4e4d089973e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0ce0917937449c6af20a4e4d089973e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34841 } }
I20260924 21:11:45.967424 20841 leader_election.cc:302] T 00000000000000000000000000000000 P a0ce0917937449c6af20a4e4d089973e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a0ce0917937449c6af20a4e4d089973e; no voters:
I20260924 21:11:45.967530 20841 leader_election.cc:288] T 00000000000000000000000000000000 P a0ce0917937449c6af20a4e4d089973e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 21:11:45.967599 20851 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a0ce0917937449c6af20a4e4d089973e [term 1 FOLLOWER]: Leader election won for term 1
I20260924 21:11:45.967712 20851 raft_consensus.cc:686] T 00000000000000000000000000000000 P a0ce0917937449c6af20a4e4d089973e [term 1 LEADER]: Becoming Leader. State: Replica: a0ce0917937449c6af20a4e4d089973e, State: Running, Role: LEADER
I20260924 21:11:45.967743 20841 sys_catalog.cc:564] T 00000000000000000000000000000000 P a0ce0917937449c6af20a4e4d089973e [sys.catalog]: configured and running, proceeding with master startup.
I20260924 21:11:45.967823 20851 consensus_queue.cc:237] T 00000000000000000000000000000000 P a0ce0917937449c6af20a4e4d089973e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0ce0917937449c6af20a4e4d089973e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34841 } }
I20260924 21:11:45.968158 20852 sys_catalog.cc:455] T 00000000000000000000000000000000 P a0ce0917937449c6af20a4e4d089973e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a0ce0917937449c6af20a4e4d089973e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0ce0917937449c6af20a4e4d089973e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34841 } } }
I20260924 21:11:45.968235 20852 sys_catalog.cc:458] T 00000000000000000000000000000000 P a0ce0917937449c6af20a4e4d089973e [sys.catalog]: This master's current role is: LEADER
I20260924 21:11:45.968393 20853 sys_catalog.cc:455] T 00000000000000000000000000000000 P a0ce0917937449c6af20a4e4d089973e [sys.catalog]: SysCatalogTable state changed. Reason: New leader a0ce0917937449c6af20a4e4d089973e. Latest consensus state: current_term: 1 leader_uuid: "a0ce0917937449c6af20a4e4d089973e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0ce0917937449c6af20a4e4d089973e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34841 } } }
I20260924 21:11:45.968480 20853 sys_catalog.cc:458] T 00000000000000000000000000000000 P a0ce0917937449c6af20a4e4d089973e [sys.catalog]: This master's current role is: LEADER
I20260924 21:11:45.968415 20859 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 21:11:45.969023 20859 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 21:11:45.970360 20859 catalog_manager.cc:1347] Generated new cluster ID: 0c0b92f3d3074b0888e59375b8a1bc0c
I20260924 21:11:45.970412 20859 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 21:11:45.977660 20859 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 21:11:45.977802 20859 catalog_manager.cc:1504] Loading token signing keys...
I20260924 21:11:45.982133 20859 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a0ce0917937449c6af20a4e4d089973e: Generated new TSK 0
I20260924 21:11:45.982275 20859 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 21:11:45.985520 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAzNNjS/build/release/bin/kudu with pid 20773
2026-09-24T21:11:45Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1147 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790284076800039-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 21:11:46 dist-test-slave-c7tk krb5kdc[20872](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 21:11:46 dist-test-slave-c7tk krb5kdc[20872](info): set up 2 sockets
Sep 24 21:11:46 dist-test-slave-c7tk krb5kdc[20872](info): commencing operation
krb5kdc: starting...
W20260924 21:11:46.937391 392 mini_kdc.cc:121] Time spent starting KDC: real 0.934s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790284076800039-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790284076800039-392-0/krb5kdc/test-user.keytab.
Sep 24 21:11:47 dist-test-slave-c7tk krb5kdc[20872](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284307, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T21:11:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T21:11:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790284076800039-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790284076800039-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 21:11:47.037405 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAzNNjS/build/release/bin/kudu
/tmp/dist-test-taskAzNNjS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790284076800039-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790284076800039-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790284076800039-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790284076800039-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45169
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790284076800039-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38133
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45169
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790284076800039-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790284076800039-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790284076800039-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 21:11:47.108966 20885 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 21:11:47.109289 20885 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 21:11:47.109328 20885 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 21:11:47.110558 20885 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 21:11:47.110622 20885 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 21:11:47.110644 20885 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 21:11:47.110663 20885 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 21:11:47.110692 20885 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 21:11:47.112140 20885 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38133
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790284076800039-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790284076800039-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45169
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790284076800039-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45169
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790284076800039-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20885
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790284076800039-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 19:49:06 UTC on ccycloud.quasar-vhqjoc.root.comops.site
build id 4670864
I20260924 21:11:47.112411 20885 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 21:11:47.112625 20885 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 21:11:47.117362 20891 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 21:11:47.117309 20890 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 21:11:47.117336 20885 server_base.cc:1034] running on GCE node
W20260924 21:11:47.117317 20893 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 21:11:47.117830 20885 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 21:11:47.118085 20885 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 21:11:47.119220 20885 hybrid_clock.cc:648] HybridClock initialized: now 1790284307119209 us; error 27 us; skew 500 ppm
Sep 24 21:11:47 dist-test-slave-c7tk krb5kdc[20872](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284307, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 21:11:47.121743 20885 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 21:11:47.122200 20885 webserver.cc:460] Webserver started at http://127.0.0.1:36689/ using document root <none> and password file <none>
I20260924 21:11:47.122381 20885 fs_manager.cc:359] Metadata directory not provided
I20260924 21:11:47.122418 20885 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 21:11:47.122509 20885 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 21:11:47.123437 20885 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790284076800039-392-0/minicluster-data/master-0/data/instance:
uuid: "ae6f863501114a45b79742ed6000e812"
format_stamp: "Formatted at 2026-09-24 21:11:47 on dist-test-slave-c7tk"
server_key: "4fe5b77383ff8eb84b459f3fd3e56f67"
server_key_iv: "704d390ce471551b3997d467f4e19a6b"
server_key_version: "encryptionkey@0"
I20260924 21:11:47.123737 20885 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790284076800039-392-0/minicluster-data/master-0/wal/instance:
uuid: "ae6f863501114a45b79742ed6000e812"
format_stamp: "Formatted at 2026-09-24 21:11:47 on dist-test-slave-c7tk"
server_key: "4fe5b77383ff8eb84b459f3fd3e56f67"
server_key_iv: "704d390ce471551b3997d467f4e19a6b"
server_key_version: "encryptionkey@0"
I20260924 21:11:47.124951 20885 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 21:11:47.125658 20900 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 21:11:47.125842 20885 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 21:11:47.125931 20885 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790284076800039-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790284076800039-392-0/minicluster-data/master-0/wal
uuid: "ae6f863501114a45b79742ed6000e812"
format_stamp: "Formatted at 2026-09-24 21:11:47 on dist-test-slave-c7tk"
server_key: "4fe5b77383ff8eb84b459f3fd3e56f67"
server_key_iv: "704d390ce471551b3997d467f4e19a6b"
server_key_version: "encryptionkey@0"
I20260924 21:11:47.125983 20885 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790284076800039-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790284076800039-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790284076800039-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 21:11:47.153937 20885 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 21:11:47.155601 20885 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 21:11:47.155699 20885 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 21:11:47.159323 20885 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45169
I20260924 21:11:47.159385 20952 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45169 every 8 connection(s)
I20260924 21:11:47.159729 20885 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790284076800039-392-0/minicluster-data/master-0/data/info.pb
I20260924 21:11:47.160565 20953 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 21:11:47.161410 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAzNNjS/build/release/bin/kudu as pid 20885
I20260924 21:11:47.161531 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790284076800039-392-0/minicluster-data/master-0/wal/instance
I20260924 21:11:47.161723 392 external_mini_cluster.cc:1428] Setting key 65cf9d59a9d5a492616fb515f9cf454d
I20260924 21:11:47.163475 20953 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ae6f863501114a45b79742ed6000e812: Bootstrap starting.
I20260924 21:11:47.163990 20953 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ae6f863501114a45b79742ed6000e812: Neither blocks nor log segments found. Creating new log.
I20260924 21:11:47.164268 20953 log.cc:824] T 00000000000000000000000000000000 P ae6f863501114a45b79742ed6000e812: Log is configured to *not* fsync() on all Append() calls
I20260924 21:11:47.164830 20953 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ae6f863501114a45b79742ed6000e812: No bootstrap required, opened a new log
I20260924 21:11:47.166567 20953 raft_consensus.cc:348] T 00000000000000000000000000000000 P ae6f863501114a45b79742ed6000e812 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae6f863501114a45b79742ed6000e812" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45169 } }
I20260924 21:11:47.166709 20953 raft_consensus.cc:374] T 00000000000000000000000000000000 P ae6f863501114a45b79742ed6000e812 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 21:11:47.166755 20953 raft_consensus.cc:729] T 00000000000000000000000000000000 P ae6f863501114a45b79742ed6000e812 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae6f863501114a45b79742ed6000e812, State: Initialized, Role: FOLLOWER
I20260924 21:11:47.166870 20953 consensus_queue.cc:260] T 00000000000000000000000000000000 P ae6f863501114a45b79742ed6000e812 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae6f863501114a45b79742ed6000e812" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45169 } }
I20260924 21:11:47.166953 20953 raft_consensus.cc:388] T 00000000000000000000000000000000 P ae6f863501114a45b79742ed6000e812 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 21:11:47.166991 20953 raft_consensus.cc:482] T 00000000000000000000000000000000 P ae6f863501114a45b79742ed6000e812 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 21:11:47.167032 20953 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ae6f863501114a45b79742ed6000e812 [term 0 FOLLOWER]: Advancing to term 1
I20260924 21:11:47.167596 20953 raft_consensus.cc:504] T 00000000000000000000000000000000 P ae6f863501114a45b79742ed6000e812 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae6f863501114a45b79742ed6000e812" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45169 } }
Sep 24 21:11:47 dist-test-slave-c7tk krb5kdc[20872](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284307, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260924 21:11:47.167694 20953 leader_election.cc:302] T 00000000000000000000000000000000 P ae6f863501114a45b79742ed6000e812 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ae6f863501114a45b79742ed6000e812; no voters:
I20260924 21:11:47.167851 20953 leader_election.cc:288] T 00000000000000000000000000000000 P ae6f863501114a45b79742ed6000e812 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 21:11:47.167882 20958 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ae6f863501114a45b79742ed6000e812 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 21:11:47.168040 20958 raft_consensus.cc:686] T 00000000000000000000000000000000 P ae6f863501114a45b79742ed6000e812 [term 1 LEADER]: Becoming Leader. State: Replica: ae6f863501114a45b79742ed6000e812, State: Running, Role: LEADER
I20260924 21:11:47.168066 20953 sys_catalog.cc:564] T 00000000000000000000000000000000 P ae6f863501114a45b79742ed6000e812 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 21:11:47.169538 20958 consensus_queue.cc:237] T 00000000000000000000000000000000 P ae6f863501114a45b79742ed6000e812 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae6f863501114a45b79742ed6000e812" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45169 } }
I20260924 21:11:47.170123 20960 sys_catalog.cc:455] T 00000000000000000000000000000000 P ae6f863501114a45b79742ed6000e812 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ae6f863501114a45b79742ed6000e812" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae6f863501114a45b79742ed6000e812" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45169 } } }
I20260924 21:11:47.170251 20960 sys_catalog.cc:458] T 00000000000000000000000000000000 P ae6f863501114a45b79742ed6000e812 [sys.catalog]: This master's current role is: LEADER
W20260924 21:11:47.170496 20972 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ae6f863501114a45b79742ed6000e812: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 21:11:47.170544 20972 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 21:11:47.170583 20959 sys_catalog.cc:455] T 00000000000000000000000000000000 P ae6f863501114a45b79742ed6000e812 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ae6f863501114a45b79742ed6000e812. Latest consensus state: current_term: 1 leader_uuid: "ae6f863501114a45b79742ed6000e812" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae6f863501114a45b79742ed6000e812" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45169 } } }
I20260924 21:11:47.170641 20959 sys_catalog.cc:458] T 00000000000000000000000000000000 P ae6f863501114a45b79742ed6000e812 [sys.catalog]: This master's current role is: LEADER
I20260924 21:11:47.170733 20974 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 21:11:47.170867 20974 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 21:11:47.171535 20956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:11:47.162238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51147 (local address 127.0.0.1:45169)
0924 21:11:47.162415 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0924 21:11:47.162424 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 21:11:47.162457 (+ 33us) server_negotiation.cc:407] Connection header received
0924 21:11:47.162851 (+ 394us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:11:47.162857 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:11:47.162904 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:11:47.163047 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:11:47.163675 (+ 628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:11:47.164584 (+ 909us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:11:47.165327 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:11:47.165715 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:11:47.165846 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:11:47.168143 (+ 2297us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 21:11:47.168146 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 21:11:47.168146 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 21:11:47.168156 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0924 21:11:47.169853 (+ 1697us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:11:47.170430 (+ 577us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:11:47.170432 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:11:47.170433 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:11:47.170525 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:11:47.170739 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:11:47.170739 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:11:47.170740 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:11:47.171025 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 21:11:47.171185 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0924 21:11:47.171244 (+ 59us) server_negotiation.cc:299] Negotiation successful
0924 21:11:47.171363 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":76,"threads_started":1}
I20260924 21:11:47.172060 20974 catalog_manager.cc:1347] Generated new cluster ID: 934c0138c25b482680401ac818431f66
I20260924 21:11:47.172104 20974 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 21:11:47.172370 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 21:11:47.180104 20956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:11:47.173758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54973 (local address 127.0.0.1:45169)
0924 21:11:47.173827 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0924 21:11:47.173831 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 21:11:47.173919 (+ 88us) server_negotiation.cc:407] Connection header received
0924 21:11:47.173976 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:11:47.173976 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:11:47.173984 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:11:47.174045 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:11:47.174568 (+ 523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:11:47.175141 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:11:47.176041 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:11:47.176328 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:11:47.176379 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:11:47.177639 (+ 1260us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 21:11:47.177640 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 21:11:47.177640 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 21:11:47.177646 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 21:11:47.178983 (+ 1337us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:11:47.179387 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:11:47.179388 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:11:47.179389 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:11:47.179438 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:11:47.179635 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:11:47.179635 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:11:47.179636 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:11:47.179742 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 21:11:47.179820 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0924 21:11:47.179958 (+ 138us) server_negotiation.cc:299] Negotiation successful
0924 21:11:47.180023 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260924 21:11:47.181427 20974 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 21:11:47.181995 20974 catalog_manager.cc:1504] Loading token signing keys...
I20260924 21:11:47.188352 20974 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ae6f863501114a45b79742ed6000e812: Generated new TSK 0
I20260924 21:11:47.188499 20974 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 21:11:47.191418 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAzNNjS/build/release/bin/kudu with pid 20885
2026-09-24T21:11:47Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1205 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790284076800039-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 21:11:47 dist-test-slave-c7tk krb5kdc[20984](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 21:11:47 dist-test-slave-c7tk krb5kdc[20984](info): set up 2 sockets
Sep 24 21:11:47 dist-test-slave-c7tk krb5kdc[20984](info): commencing operation
krb5kdc: starting...
W20260924 21:11:48.117690 392 mini_kdc.cc:121] Time spent starting KDC: real 0.910s user 0.000s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790284076800039-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790284076800039-392-0/krb5kdc/test-user.keytab.
Sep 24 21:11:48 dist-test-slave-c7tk krb5kdc[20984](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284308, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T21:11:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T21:11:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790284076800039-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790284076800039-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 21:11:48.220384 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAzNNjS/build/release/bin/kudu
/tmp/dist-test-taskAzNNjS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790284076800039-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790284076800039-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790284076800039-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790284076800039-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43877
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790284076800039-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37241
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43877
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790284076800039-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790284076800039-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790284076800039-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 21:11:48.290823 20997 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 21:11:48.290988 20997 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 21:11:48.291007 20997 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 21:11:48.292049 20997 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 21:11:48.292088 20997 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 21:11:48.292102 20997 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 21:11:48.292114 20997 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 21:11:48.292164 20997 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 21:11:48.293522 20997 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37241
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790284076800039-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790284076800039-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43877
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790284076800039-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43877
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790284076800039-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20997
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790284076800039-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 19:49:06 UTC on ccycloud.quasar-vhqjoc.root.comops.site
build id 4670864
I20260924 21:11:48.293740 20997 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 21:11:48.293994 20997 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 21:11:48.298157 21003 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 21:11:48.298205 21002 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 21:11:48.298161 21005 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 21:11:48.298316 20997 server_base.cc:1034] running on GCE node
I20260924 21:11:48.298513 20997 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 21:11:48.298760 20997 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 21:11:48.299916 20997 hybrid_clock.cc:648] HybridClock initialized: now 1790284308299903 us; error 39 us; skew 500 ppm
Sep 24 21:11:48 dist-test-slave-c7tk krb5kdc[20984](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284308, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 21:11:48.302018 20997 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 21:11:48.302352 20997 webserver.cc:460] Webserver started at http://127.0.0.1:43501/ using document root <none> and password file <none>
I20260924 21:11:48.302541 20997 fs_manager.cc:359] Metadata directory not provided
I20260924 21:11:48.302598 20997 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 21:11:48.302709 20997 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 21:11:48.303606 20997 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790284076800039-392-0/minicluster-data/master-0/data/instance:
uuid: "b42422121cce4efd8a64ddc1700f0385"
format_stamp: "Formatted at 2026-09-24 21:11:48 on dist-test-slave-c7tk"
server_key: "f542b2dfc109c4e18c468b308da65be1"
server_key_iv: "b85e0501bd13c7cc520ba915f6c40eb4"
server_key_version: "encryptionkey@0"
I20260924 21:11:48.303905 20997 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790284076800039-392-0/minicluster-data/master-0/wal/instance:
uuid: "b42422121cce4efd8a64ddc1700f0385"
format_stamp: "Formatted at 2026-09-24 21:11:48 on dist-test-slave-c7tk"
server_key: "f542b2dfc109c4e18c468b308da65be1"
server_key_iv: "b85e0501bd13c7cc520ba915f6c40eb4"
server_key_version: "encryptionkey@0"
I20260924 21:11:48.305040 20997 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 21:11:48.305610 21012 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 21:11:48.305755 20997 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 21:11:48.305856 20997 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790284076800039-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790284076800039-392-0/minicluster-data/master-0/wal
uuid: "b42422121cce4efd8a64ddc1700f0385"
format_stamp: "Formatted at 2026-09-24 21:11:48 on dist-test-slave-c7tk"
server_key: "f542b2dfc109c4e18c468b308da65be1"
server_key_iv: "b85e0501bd13c7cc520ba915f6c40eb4"
server_key_version: "encryptionkey@0"
I20260924 21:11:48.305928 20997 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790284076800039-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790284076800039-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790284076800039-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 21:11:48.342893 20997 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 21:11:48.344543 20997 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 21:11:48.344687 20997 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 21:11:48.348706 20997 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43877
I20260924 21:11:48.348764 21064 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43877 every 8 connection(s)
I20260924 21:11:48.349103 20997 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790284076800039-392-0/minicluster-data/master-0/data/info.pb
I20260924 21:11:48.349996 21065 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 21:11:48.352067 21065 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b42422121cce4efd8a64ddc1700f0385: Bootstrap starting.
I20260924 21:11:48.352648 21065 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b42422121cce4efd8a64ddc1700f0385: Neither blocks nor log segments found. Creating new log.
I20260924 21:11:48.352948 21065 log.cc:824] T 00000000000000000000000000000000 P b42422121cce4efd8a64ddc1700f0385: Log is configured to *not* fsync() on all Append() calls
I20260924 21:11:48.353626 21065 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b42422121cce4efd8a64ddc1700f0385: No bootstrap required, opened a new log
I20260924 21:11:48.353991 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAzNNjS/build/release/bin/kudu as pid 20997
I20260924 21:11:48.354126 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790284076800039-392-0/minicluster-data/master-0/wal/instance
I20260924 21:11:48.354323 392 external_mini_cluster.cc:1428] Setting key df6898f5eb23eecba66ca11aa78c71cb
I20260924 21:11:48.355566 21065 raft_consensus.cc:348] T 00000000000000000000000000000000 P b42422121cce4efd8a64ddc1700f0385 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b42422121cce4efd8a64ddc1700f0385" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43877 } }
I20260924 21:11:48.355723 21065 raft_consensus.cc:374] T 00000000000000000000000000000000 P b42422121cce4efd8a64ddc1700f0385 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 21:11:48.355762 21065 raft_consensus.cc:729] T 00000000000000000000000000000000 P b42422121cce4efd8a64ddc1700f0385 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b42422121cce4efd8a64ddc1700f0385, State: Initialized, Role: FOLLOWER
I20260924 21:11:48.355940 21065 consensus_queue.cc:260] T 00000000000000000000000000000000 P b42422121cce4efd8a64ddc1700f0385 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b42422121cce4efd8a64ddc1700f0385" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43877 } }
I20260924 21:11:48.356040 21065 raft_consensus.cc:388] T 00000000000000000000000000000000 P b42422121cce4efd8a64ddc1700f0385 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 21:11:48.356087 21065 raft_consensus.cc:482] T 00000000000000000000000000000000 P b42422121cce4efd8a64ddc1700f0385 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 21:11:48.356137 21065 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b42422121cce4efd8a64ddc1700f0385 [term 0 FOLLOWER]: Advancing to term 1
I20260924 21:11:48.356664 21065 raft_consensus.cc:504] T 00000000000000000000000000000000 P b42422121cce4efd8a64ddc1700f0385 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b42422121cce4efd8a64ddc1700f0385" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43877 } }
I20260924 21:11:48.356741 21065 leader_election.cc:302] T 00000000000000000000000000000000 P b42422121cce4efd8a64ddc1700f0385 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b42422121cce4efd8a64ddc1700f0385; no voters:
I20260924 21:11:48.356956 21065 leader_election.cc:288] T 00000000000000000000000000000000 P b42422121cce4efd8a64ddc1700f0385 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 21:11:48.356984 21070 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b42422121cce4efd8a64ddc1700f0385 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 21:11:48.357100 21070 raft_consensus.cc:686] T 00000000000000000000000000000000 P b42422121cce4efd8a64ddc1700f0385 [term 1 LEADER]: Becoming Leader. State: Replica: b42422121cce4efd8a64ddc1700f0385, State: Running, Role: LEADER
I20260924 21:11:48.357215 21065 sys_catalog.cc:564] T 00000000000000000000000000000000 P b42422121cce4efd8a64ddc1700f0385 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 21:11:48.357226 21070 consensus_queue.cc:237] T 00000000000000000000000000000000 P b42422121cce4efd8a64ddc1700f0385 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b42422121cce4efd8a64ddc1700f0385" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43877 } }
I20260924 21:11:48.357671 21071 sys_catalog.cc:455] T 00000000000000000000000000000000 P b42422121cce4efd8a64ddc1700f0385 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b42422121cce4efd8a64ddc1700f0385" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b42422121cce4efd8a64ddc1700f0385" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43877 } } }
I20260924 21:11:48.357788 21071 sys_catalog.cc:458] T 00000000000000000000000000000000 P b42422121cce4efd8a64ddc1700f0385 [sys.catalog]: This master's current role is: LEADER
I20260924 21:11:48.357689 21072 sys_catalog.cc:455] T 00000000000000000000000000000000 P b42422121cce4efd8a64ddc1700f0385 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b42422121cce4efd8a64ddc1700f0385. Latest consensus state: current_term: 1 leader_uuid: "b42422121cce4efd8a64ddc1700f0385" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b42422121cce4efd8a64ddc1700f0385" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43877 } } }
I20260924 21:11:48.357955 21072 sys_catalog.cc:458] T 00000000000000000000000000000000 P b42422121cce4efd8a64ddc1700f0385 [sys.catalog]: This master's current role is: LEADER
W20260924 21:11:48.359089 21086 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b42422121cce4efd8a64ddc1700f0385: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 21:11:48.359201 21086 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 21:11:48.359289 21079 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 21:11:48.359457 21079 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 21:11:48.360531 21069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:11:48.354814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45107 (local address 127.0.0.1:43877)
0924 21:11:48.354990 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0924 21:11:48.354997 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 21:11:48.355022 (+ 25us) server_negotiation.cc:407] Connection header received
0924 21:11:48.355554 (+ 532us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:11:48.355560 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:11:48.355605 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:11:48.355713 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:11:48.355982 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:11:48.357387 (+ 1405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:11:48.358480 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:11:48.359073 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:11:48.359204 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:11:48.359811 (+ 607us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 21:11:48.359815 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 21:11:48.359815 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 21:11:48.360036 (+ 221us) server_negotiation.cc:922] calling sasl_server_start()
0924 21:11:48.360073 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 21:11:48.360087 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 21:11:48.360153 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0924 21:11:48.360256 (+ 103us) server_negotiation.cc:299] Negotiation successful
0924 21:11:48.360327 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":88,"threads_started":1}
I20260924 21:11:48.361025 21079 catalog_manager.cc:1347] Generated new cluster ID: 775dca24c7f040bfa98fcebbb2da106c
I20260924 21:11:48.361078 21079 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 21:11:48.361573 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 21:11:48.365437 21069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:11:48.362921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49835 (local address 127.0.0.1:43877)
0924 21:11:48.362952 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0924 21:11:48.362954 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 21:11:48.363074 (+ 120us) server_negotiation.cc:407] Connection header received
0924 21:11:48.363138 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:11:48.363139 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:11:48.363147 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:11:48.363196 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:11:48.363288 (+ 92us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:11:48.363698 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:11:48.364354 (+ 656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:11:48.364538 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:11:48.364618 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:11:48.365227 (+ 609us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 21:11:48.365228 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 21:11:48.365229 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 21:11:48.365233 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 21:11:48.365244 (+ 11us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 21:11:48.365247 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 21:11:48.365289 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0924 21:11:48.365337 (+ 48us) server_negotiation.cc:299] Negotiation successful
0924 21:11:48.365355 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260924 21:11:48.376986 21079 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 21:11:48.377568 21079 catalog_manager.cc:1504] Loading token signing keys...
I20260924 21:11:48.382714 21079 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b42422121cce4efd8a64ddc1700f0385: Generated new TSK 0
I20260924 21:11:48.382845 21079 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 21:11:48.389104 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAzNNjS/build/release/bin/kudu with pid 20997
2026-09-24T21:11:48Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1198 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790284076800039-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 21:11:48 dist-test-slave-c7tk krb5kdc[21096](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 21:11:48 dist-test-slave-c7tk krb5kdc[21096](info): set up 2 sockets
Sep 24 21:11:48 dist-test-slave-c7tk krb5kdc[21096](info): commencing operation
krb5kdc: starting...
W20260924 21:11:49.317395 392 mini_kdc.cc:121] Time spent starting KDC: real 0.911s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790284076800039-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790284076800039-392-0/krb5kdc/test-user.keytab.
Sep 24 21:11:49 dist-test-slave-c7tk krb5kdc[21096](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284309, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T21:11:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T21:11:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790284076800039-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790284076800039-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 21:11:49.421732 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAzNNjS/build/release/bin/kudu
/tmp/dist-test-taskAzNNjS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790284076800039-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790284076800039-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790284076800039-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790284076800039-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46549
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790284076800039-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34087
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46549
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790284076800039-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790284076800039-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790284076800039-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 21:11:49.491600 21109 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 21:11:49.491755 21109 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 21:11:49.491772 21109 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 21:11:49.492786 21109 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 21:11:49.492825 21109 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 21:11:49.492838 21109 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 21:11:49.492849 21109 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 21:11:49.492884 21109 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 21:11:49.494269 21109 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34087
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790284076800039-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790284076800039-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46549
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790284076800039-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46549
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790284076800039-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21109
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790284076800039-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 19:49:06 UTC on ccycloud.quasar-vhqjoc.root.comops.site
build id 4670864
I20260924 21:11:49.494474 21109 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 21:11:49.494653 21109 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 21:11:49.498842 21115 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 21:11:49.498874 21109 server_base.cc:1034] running on GCE node
W20260924 21:11:49.498963 21117 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 21:11:49.498824 21114 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 21:11:49.499248 21109 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 21:11:49.499478 21109 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 21:11:49.500609 21109 hybrid_clock.cc:648] HybridClock initialized: now 1790284309500589 us; error 36 us; skew 500 ppm
Sep 24 21:11:49 dist-test-slave-c7tk krb5kdc[21096](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284309, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 21:11:49.502655 21109 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 21:11:49.503053 21109 webserver.cc:460] Webserver started at http://127.0.0.1:40369/ using document root <none> and password file <none>
I20260924 21:11:49.503255 21109 fs_manager.cc:359] Metadata directory not provided
I20260924 21:11:49.503297 21109 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 21:11:49.503438 21109 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 21:11:49.504352 21109 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790284076800039-392-0/minicluster-data/master-0/data/instance:
uuid: "55e8dad2cb554b81846ff393e87de094"
format_stamp: "Formatted at 2026-09-24 21:11:49 on dist-test-slave-c7tk"
server_key: "d22f3a4f1e98e5888692644572b36509"
server_key_iv: "31eea956b188eb698ee17e3c14b791f8"
server_key_version: "encryptionkey@0"
I20260924 21:11:49.504693 21109 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790284076800039-392-0/minicluster-data/master-0/wal/instance:
uuid: "55e8dad2cb554b81846ff393e87de094"
format_stamp: "Formatted at 2026-09-24 21:11:49 on dist-test-slave-c7tk"
server_key: "d22f3a4f1e98e5888692644572b36509"
server_key_iv: "31eea956b188eb698ee17e3c14b791f8"
server_key_version: "encryptionkey@0"
I20260924 21:11:49.505874 21109 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 21:11:49.506529 21124 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 21:11:49.506687 21109 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 21:11:49.506774 21109 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790284076800039-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790284076800039-392-0/minicluster-data/master-0/wal
uuid: "55e8dad2cb554b81846ff393e87de094"
format_stamp: "Formatted at 2026-09-24 21:11:49 on dist-test-slave-c7tk"
server_key: "d22f3a4f1e98e5888692644572b36509"
server_key_iv: "31eea956b188eb698ee17e3c14b791f8"
server_key_version: "encryptionkey@0"
I20260924 21:11:49.506851 21109 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790284076800039-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790284076800039-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790284076800039-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 21:11:49.509845 21109 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 21:11:49.511440 21109 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 21:11:49.511586 21109 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 21:11:49.515158 21109 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46549
I20260924 21:11:49.515182 21176 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46549 every 8 connection(s)
I20260924 21:11:49.515442 21109 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790284076800039-392-0/minicluster-data/master-0/data/info.pb
I20260924 21:11:49.516266 21177 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 21:11:49.518122 21177 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 55e8dad2cb554b81846ff393e87de094: Bootstrap starting.
I20260924 21:11:49.518646 21177 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 55e8dad2cb554b81846ff393e87de094: Neither blocks nor log segments found. Creating new log.
I20260924 21:11:49.518855 21177 log.cc:824] T 00000000000000000000000000000000 P 55e8dad2cb554b81846ff393e87de094: Log is configured to *not* fsync() on all Append() calls
I20260924 21:11:49.519510 21177 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 55e8dad2cb554b81846ff393e87de094: No bootstrap required, opened a new log
I20260924 21:11:49.520903 21177 raft_consensus.cc:348] T 00000000000000000000000000000000 P 55e8dad2cb554b81846ff393e87de094 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55e8dad2cb554b81846ff393e87de094" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46549 } }
I20260924 21:11:49.521037 21177 raft_consensus.cc:374] T 00000000000000000000000000000000 P 55e8dad2cb554b81846ff393e87de094 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 21:11:49.521059 21177 raft_consensus.cc:729] T 00000000000000000000000000000000 P 55e8dad2cb554b81846ff393e87de094 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 55e8dad2cb554b81846ff393e87de094, State: Initialized, Role: FOLLOWER
I20260924 21:11:49.521170 21177 consensus_queue.cc:260] T 00000000000000000000000000000000 P 55e8dad2cb554b81846ff393e87de094 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55e8dad2cb554b81846ff393e87de094" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46549 } }
I20260924 21:11:49.521239 21177 raft_consensus.cc:388] T 00000000000000000000000000000000 P 55e8dad2cb554b81846ff393e87de094 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 21:11:49.521258 21177 raft_consensus.cc:482] T 00000000000000000000000000000000 P 55e8dad2cb554b81846ff393e87de094 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 21:11:49.521283 21177 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 55e8dad2cb554b81846ff393e87de094 [term 0 FOLLOWER]: Advancing to term 1
I20260924 21:11:49.521682 21177 raft_consensus.cc:504] T 00000000000000000000000000000000 P 55e8dad2cb554b81846ff393e87de094 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55e8dad2cb554b81846ff393e87de094" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46549 } }
I20260924 21:11:49.521751 21177 leader_election.cc:302] T 00000000000000000000000000000000 P 55e8dad2cb554b81846ff393e87de094 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 55e8dad2cb554b81846ff393e87de094; no voters:
I20260924 21:11:49.521900 21177 leader_election.cc:288] T 00000000000000000000000000000000 P 55e8dad2cb554b81846ff393e87de094 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 21:11:49.521960 21180 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 55e8dad2cb554b81846ff393e87de094 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 21:11:49.522096 21180 raft_consensus.cc:686] T 00000000000000000000000000000000 P 55e8dad2cb554b81846ff393e87de094 [term 1 LEADER]: Becoming Leader. State: Replica: 55e8dad2cb554b81846ff393e87de094, State: Running, Role: LEADER
I20260924 21:11:49.522145 21177 sys_catalog.cc:564] T 00000000000000000000000000000000 P 55e8dad2cb554b81846ff393e87de094 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 21:11:49.522174 21180 consensus_queue.cc:237] T 00000000000000000000000000000000 P 55e8dad2cb554b81846ff393e87de094 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55e8dad2cb554b81846ff393e87de094" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46549 } }
I20260924 21:11:49.522591 21182 sys_catalog.cc:455] T 00000000000000000000000000000000 P 55e8dad2cb554b81846ff393e87de094 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 55e8dad2cb554b81846ff393e87de094. Latest consensus state: current_term: 1 leader_uuid: "55e8dad2cb554b81846ff393e87de094" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55e8dad2cb554b81846ff393e87de094" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46549 } } }
I20260924 21:11:49.522680 21181 sys_catalog.cc:455] T 00000000000000000000000000000000 P 55e8dad2cb554b81846ff393e87de094 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "55e8dad2cb554b81846ff393e87de094" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55e8dad2cb554b81846ff393e87de094" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46549 } } }
I20260924 21:11:49.522727 21182 sys_catalog.cc:458] T 00000000000000000000000000000000 P 55e8dad2cb554b81846ff393e87de094 [sys.catalog]: This master's current role is: LEADER
I20260924 21:11:49.522785 21181 sys_catalog.cc:458] T 00000000000000000000000000000000 P 55e8dad2cb554b81846ff393e87de094 [sys.catalog]: This master's current role is: LEADER
I20260924 21:11:49.523154 21189 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 21:11:49.523671 21189 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 21:11:49.524925 21189 catalog_manager.cc:1347] Generated new cluster ID: 5f1b2d09c1894c3596143b4bfa410664
I20260924 21:11:49.524971 21189 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 21:11:49.525061 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAzNNjS/build/release/bin/kudu as pid 21109
I20260924 21:11:49.525164 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790284076800039-392-0/minicluster-data/master-0/wal/instance
I20260924 21:11:49.525326 392 external_mini_cluster.cc:1428] Setting key f805106534b2cfa2acb84e6f58994f23
I20260924 21:11:49.527384 21200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:11:49.525844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36317 (local address 127.0.0.1:46549)
0924 21:11:49.526039 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0924 21:11:49.526047 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 21:11:49.526078 (+ 31us) server_negotiation.cc:407] Connection header received
0924 21:11:49.526172 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:11:49.526177 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:11:49.526204 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:11:49.526327 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:11:49.527021 (+ 694us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 21:11:49.527025 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 21:11:49.527026 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 21:11:49.527029 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0924 21:11:49.527065 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 21:11:49.527079 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 21:11:49.527145 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0924 21:11:49.527202 (+ 57us) server_negotiation.cc:299] Negotiation successful
0924 21:11:49.527237 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":154,"thread_start_us":58,"threads_started":1}
I20260924 21:11:49.528146 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 21:11:49.530747 21200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:11:49.529562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50455 (local address 127.0.0.1:46549)
0924 21:11:49.529591 (+ 29us) server_negotiation.cc:206] Beginning negotiation
0924 21:11:49.529593 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 21:11:49.529716 (+ 123us) server_negotiation.cc:407] Connection header received
0924 21:11:49.529775 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:11:49.529776 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:11:49.529780 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:11:49.529859 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:11:49.530557 (+ 698us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 21:11:49.530558 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 21:11:49.530558 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 21:11:49.530562 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 21:11:49.530574 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 21:11:49.530577 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 21:11:49.530627 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0924 21:11:49.530687 (+ 60us) server_negotiation.cc:299] Negotiation successful
0924 21:11:49.530697 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260924 21:11:49.543931 21189 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 21:11:49.544070 21189 catalog_manager.cc:1504] Loading token signing keys...
I20260924 21:11:49.552131 21189 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 55e8dad2cb554b81846ff393e87de094: Generated new TSK 0
I20260924 21:11:49.552280 21189 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 21:11:49.554685 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAzNNjS/build/release/bin/kudu with pid 21109
2026-09-24T21:11:49Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1165 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790284076800039-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 21:11:49 dist-test-slave-c7tk krb5kdc[21208](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 21:11:49 dist-test-slave-c7tk krb5kdc[21208](info): set up 2 sockets
Sep 24 21:11:49 dist-test-slave-c7tk krb5kdc[21208](info): commencing operation
krb5kdc: starting...
W20260924 21:11:50.481457 392 mini_kdc.cc:121] Time spent starting KDC: real 0.909s user 0.000s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790284076800039-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790284076800039-392-0/krb5kdc/test-user.keytab.
Sep 24 21:11:50 dist-test-slave-c7tk krb5kdc[21208](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284310, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T21:11:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T21:11:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790284076800039-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790284076800039-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 21:11:50.582782 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAzNNjS/build/release/bin/kudu
/tmp/dist-test-taskAzNNjS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790284076800039-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790284076800039-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790284076800039-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790284076800039-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42595
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790284076800039-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38129
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42595
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790284076800039-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790284076800039-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790284076800039-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 21:11:50.654361 21221 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 21:11:50.654577 21221 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 21:11:50.654611 21221 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 21:11:50.655635 21221 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 21:11:50.655695 21221 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 21:11:50.655717 21221 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 21:11:50.655747 21221 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 21:11:50.655772 21221 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 21:11:50.657080 21221 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38129
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790284076800039-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790284076800039-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42595
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790284076800039-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42595
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790284076800039-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21221
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790284076800039-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 19:49:06 UTC on ccycloud.quasar-vhqjoc.root.comops.site
build id 4670864
I20260924 21:11:50.657316 21221 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 21:11:50.657526 21221 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 21:11:50.662087 21227 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 21:11:50.662091 21226 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 21:11:50.662102 21229 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 21:11:50.662493 21221 server_base.cc:1034] running on GCE node
I20260924 21:11:50.662681 21221 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 21:11:50.662977 21221 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 21:11:50.664119 21221 hybrid_clock.cc:648] HybridClock initialized: now 1790284310664111 us; error 31 us; skew 500 ppm
Sep 24 21:11:50 dist-test-slave-c7tk krb5kdc[21208](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284310, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 21:11:50.672183 21221 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 21:11:50.672626 21221 webserver.cc:460] Webserver started at http://127.0.0.1:44261/ using document root <none> and password file <none>
I20260924 21:11:50.672860 21221 fs_manager.cc:359] Metadata directory not provided
I20260924 21:11:50.672946 21221 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 21:11:50.673098 21221 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 21:11:50.674185 21221 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790284076800039-392-0/minicluster-data/master-0/data/instance:
uuid: "0e9504c4d9254e9399d9b994dd32ba27"
format_stamp: "Formatted at 2026-09-24 21:11:50 on dist-test-slave-c7tk"
server_key: "d78d41e57ac642edd9eedac5dc9a3126"
server_key_iv: "ecc60b34ff9a0f54d045df9de271554c"
server_key_version: "encryptionkey@0"
I20260924 21:11:50.674510 21221 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790284076800039-392-0/minicluster-data/master-0/wal/instance:
uuid: "0e9504c4d9254e9399d9b994dd32ba27"
format_stamp: "Formatted at 2026-09-24 21:11:50 on dist-test-slave-c7tk"
server_key: "d78d41e57ac642edd9eedac5dc9a3126"
server_key_iv: "ecc60b34ff9a0f54d045df9de271554c"
server_key_version: "encryptionkey@0"
I20260924 21:11:50.675689 21221 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 21:11:50.676378 21236 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 21:11:50.676535 21221 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 21:11:50.676630 21221 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790284076800039-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790284076800039-392-0/minicluster-data/master-0/wal
uuid: "0e9504c4d9254e9399d9b994dd32ba27"
format_stamp: "Formatted at 2026-09-24 21:11:50 on dist-test-slave-c7tk"
server_key: "d78d41e57ac642edd9eedac5dc9a3126"
server_key_iv: "ecc60b34ff9a0f54d045df9de271554c"
server_key_version: "encryptionkey@0"
I20260924 21:11:50.676694 21221 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790284076800039-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790284076800039-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790284076800039-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 21:11:50.688105 21221 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 21:11:50.690215 21221 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 21:11:50.690371 21221 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 21:11:50.694064 21221 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42595
I20260924 21:11:50.694090 21288 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42595 every 8 connection(s)
I20260924 21:11:50.694380 21221 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790284076800039-392-0/minicluster-data/master-0/data/info.pb
I20260924 21:11:50.695071 21289 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 21:11:50.696218 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAzNNjS/build/release/bin/kudu as pid 21221
I20260924 21:11:50.696326 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790284076800039-392-0/minicluster-data/master-0/wal/instance
I20260924 21:11:50.696478 392 external_mini_cluster.cc:1428] Setting key fda76bcf50ec68c7f3c4f0eff6b01b0c
I20260924 21:11:50.697386 21289 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0e9504c4d9254e9399d9b994dd32ba27: Bootstrap starting.
I20260924 21:11:50.698009 21289 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0e9504c4d9254e9399d9b994dd32ba27: Neither blocks nor log segments found. Creating new log.
I20260924 21:11:50.698290 21289 log.cc:824] T 00000000000000000000000000000000 P 0e9504c4d9254e9399d9b994dd32ba27: Log is configured to *not* fsync() on all Append() calls
I20260924 21:11:50.698853 21289 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0e9504c4d9254e9399d9b994dd32ba27: No bootstrap required, opened a new log
I20260924 21:11:50.700968 21289 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0e9504c4d9254e9399d9b994dd32ba27 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e9504c4d9254e9399d9b994dd32ba27" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42595 } }
I20260924 21:11:50.701108 21289 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0e9504c4d9254e9399d9b994dd32ba27 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 21:11:50.701170 21289 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0e9504c4d9254e9399d9b994dd32ba27 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e9504c4d9254e9399d9b994dd32ba27, State: Initialized, Role: FOLLOWER
I20260924 21:11:50.701277 21289 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0e9504c4d9254e9399d9b994dd32ba27 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e9504c4d9254e9399d9b994dd32ba27" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42595 } }
I20260924 21:11:50.701371 21289 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0e9504c4d9254e9399d9b994dd32ba27 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 21:11:50.701404 21289 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0e9504c4d9254e9399d9b994dd32ba27 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 21:11:50.701442 21289 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0e9504c4d9254e9399d9b994dd32ba27 [term 0 FOLLOWER]: Advancing to term 1
I20260924 21:11:50.702137 21289 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0e9504c4d9254e9399d9b994dd32ba27 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e9504c4d9254e9399d9b994dd32ba27" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42595 } }
I20260924 21:11:50.702240 21289 leader_election.cc:302] T 00000000000000000000000000000000 P 0e9504c4d9254e9399d9b994dd32ba27 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0e9504c4d9254e9399d9b994dd32ba27; no voters:
I20260924 21:11:50.702373 21289 leader_election.cc:288] T 00000000000000000000000000000000 P 0e9504c4d9254e9399d9b994dd32ba27 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 21:11:50.702395 21294 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0e9504c4d9254e9399d9b994dd32ba27 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 21:11:50.702498 21294 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0e9504c4d9254e9399d9b994dd32ba27 [term 1 LEADER]: Becoming Leader. State: Replica: 0e9504c4d9254e9399d9b994dd32ba27, State: Running, Role: LEADER
I20260924 21:11:50.702608 21289 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0e9504c4d9254e9399d9b994dd32ba27 [sys.catalog]: configured and running, proceeding with master startup.
Sep 24 21:11:50 dist-test-slave-c7tk krb5kdc[21208](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284310, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260924 21:11:50.702610 21294 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0e9504c4d9254e9399d9b994dd32ba27 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e9504c4d9254e9399d9b994dd32ba27" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42595 } }
I20260924 21:11:50.704187 21295 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0e9504c4d9254e9399d9b994dd32ba27 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0e9504c4d9254e9399d9b994dd32ba27" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e9504c4d9254e9399d9b994dd32ba27" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42595 } } }
I20260924 21:11:50.704285 21295 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0e9504c4d9254e9399d9b994dd32ba27 [sys.catalog]: This master's current role is: LEADER
I20260924 21:11:50.704619 21296 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0e9504c4d9254e9399d9b994dd32ba27 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0e9504c4d9254e9399d9b994dd32ba27. Latest consensus state: current_term: 1 leader_uuid: "0e9504c4d9254e9399d9b994dd32ba27" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e9504c4d9254e9399d9b994dd32ba27" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42595 } } }
I20260924 21:11:50.704638 21308 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 21:11:50.704684 21296 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0e9504c4d9254e9399d9b994dd32ba27 [sys.catalog]: This master's current role is: LEADER
I20260924 21:11:50.705066 21308 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 21:11:50.706518 21308 catalog_manager.cc:1347] Generated new cluster ID: 242aa29168874c729c5546c9d9f303f7
I20260924 21:11:50.706559 21292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:11:50.696907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.21:43859 (local address 127.0.0.1:42595)
0924 21:11:50.697071 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0924 21:11:50.697078 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 21:11:50.697100 (+ 22us) server_negotiation.cc:407] Connection header received
0924 21:11:50.697474 (+ 374us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:11:50.697481 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:11:50.697534 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:11:50.697635 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:11:50.698398 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:11:50.699200 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:11:50.700264 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:11:50.700666 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:11:50.700741 (+ 75us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:11:50.703265 (+ 2524us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 21:11:50.703270 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 21:11:50.703270 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 21:11:50.703282 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 21:11:50.705131 (+ 1849us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:11:50.705490 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:11:50.705492 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:11:50.705493 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:11:50.705579 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:11:50.705811 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:11:50.705812 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:11:50.705812 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:11:50.706053 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 21:11:50.706171 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0924 21:11:50.706306 (+ 135us) server_negotiation.cc:299] Negotiation successful
0924 21:11:50.706390 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119,"thread_start_us":90,"threads_started":1}
I20260924 21:11:50.706612 21308 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 21:11:50.707351 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 21:11:50.716341 21292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:11:50.710052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.21:57261 (local address 127.0.0.1:42595)
0924 21:11:50.710084 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0924 21:11:50.710086 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 21:11:50.710256 (+ 170us) server_negotiation.cc:407] Connection header received
0924 21:11:50.710326 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:11:50.710326 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:11:50.710334 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:11:50.710384 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:11:50.710985 (+ 601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:11:50.711581 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:11:50.712425 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:11:50.712716 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:11:50.712784 (+ 68us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:11:50.714056 (+ 1272us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 21:11:50.714057 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 21:11:50.714057 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 21:11:50.714062 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 21:11:50.715276 (+ 1214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:11:50.715630 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:11:50.715630 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:11:50.715631 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:11:50.715674 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:11:50.715865 (+ 191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:11:50.715865 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:11:50.715865 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:11:50.715955 (+ 90us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 21:11:50.716025 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0924 21:11:50.716213 (+ 188us) server_negotiation.cc:299] Negotiation successful
0924 21:11:50.716270 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 21:11:50.726692 21308 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 21:11:50.727097 21308 catalog_manager.cc:1504] Loading token signing keys...
I20260924 21:11:50.733319 21308 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0e9504c4d9254e9399d9b994dd32ba27: Generated new TSK 0
I20260924 21:11:50.733492 21308 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 21:11:50.735376 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAzNNjS/build/release/bin/kudu with pid 21221
W20260924 21:11:50.739046 21206 connection.cc:535] client connection to 127.0.0.1:42595 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:42595): Connection reset by peer (error 104)
2026-09-24T21:11:50Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1180 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790284076800039-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 21:11:50 dist-test-slave-c7tk krb5kdc[21320](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 21:11:50 dist-test-slave-c7tk krb5kdc[21320](info): set up 2 sockets
Sep 24 21:11:50 dist-test-slave-c7tk krb5kdc[21320](info): commencing operation
krb5kdc: starting...
W20260924 21:11:51.657301 392 mini_kdc.cc:121] Time spent starting KDC: real 0.905s user 0.000s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790284076800039-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790284076800039-392-0/krb5kdc/test-user.keytab.
Sep 24 21:11:51 dist-test-slave-c7tk krb5kdc[21320](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284311, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T21:11:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T21:11:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790284076800039-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790284076800039-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 21:11:51.758255 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAzNNjS/build/release/bin/kudu
/tmp/dist-test-taskAzNNjS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790284076800039-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790284076800039-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790284076800039-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790284076800039-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39691
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790284076800039-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36465
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39691
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790284076800039-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790284076800039-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790284076800039-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 21:11:51.827627 21333 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 21:11:51.827787 21333 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 21:11:51.827806 21333 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 21:11:51.828819 21333 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 21:11:51.828855 21333 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 21:11:51.828868 21333 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 21:11:51.828915 21333 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 21:11:51.828939 21333 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 21:11:51.830344 21333 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36465
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790284076800039-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790284076800039-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39691
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790284076800039-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39691
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790284076800039-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21333
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790284076800039-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 19:49:06 UTC on ccycloud.quasar-vhqjoc.root.comops.site
build id 4670864
I20260924 21:11:51.830561 21333 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 21:11:51.830801 21333 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 21:11:51.835075 21339 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 21:11:51.835148 21333 server_base.cc:1034] running on GCE node
W20260924 21:11:51.835098 21341 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 21:11:51.835111 21338 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 21:11:51.835439 21333 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 21:11:51.835634 21333 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 21:11:51.836760 21333 hybrid_clock.cc:648] HybridClock initialized: now 1790284311836753 us; error 23 us; skew 500 ppm
Sep 24 21:11:51 dist-test-slave-c7tk krb5kdc[21320](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284311, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 21:11:51.838961 21333 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 21:11:51.839341 21333 webserver.cc:460] Webserver started at http://127.0.0.1:33199/ using document root <none> and password file <none>
I20260924 21:11:51.839510 21333 fs_manager.cc:359] Metadata directory not provided
I20260924 21:11:51.839568 21333 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 21:11:51.839645 21333 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 21:11:51.840508 21333 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790284076800039-392-0/minicluster-data/master-0/data/instance:
uuid: "c50343ab2c924f66ab22b75f46f118ac"
format_stamp: "Formatted at 2026-09-24 21:11:51 on dist-test-slave-c7tk"
server_key: "afa31aded125b8df6101fabf3fd11102"
server_key_iv: "43ab91f472b356b80e8ada21266c1c39"
server_key_version: "encryptionkey@0"
I20260924 21:11:51.840802 21333 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790284076800039-392-0/minicluster-data/master-0/wal/instance:
uuid: "c50343ab2c924f66ab22b75f46f118ac"
format_stamp: "Formatted at 2026-09-24 21:11:51 on dist-test-slave-c7tk"
server_key: "afa31aded125b8df6101fabf3fd11102"
server_key_iv: "43ab91f472b356b80e8ada21266c1c39"
server_key_version: "encryptionkey@0"
I20260924 21:11:51.841853 21333 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260924 21:11:51.842425 21348 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 21:11:51.842564 21333 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 21:11:51.842648 21333 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790284076800039-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790284076800039-392-0/minicluster-data/master-0/wal
uuid: "c50343ab2c924f66ab22b75f46f118ac"
format_stamp: "Formatted at 2026-09-24 21:11:51 on dist-test-slave-c7tk"
server_key: "afa31aded125b8df6101fabf3fd11102"
server_key_iv: "43ab91f472b356b80e8ada21266c1c39"
server_key_version: "encryptionkey@0"
I20260924 21:11:51.842715 21333 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790284076800039-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790284076800039-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790284076800039-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 21:11:51.851223 21333 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 21:11:51.853229 21333 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 21:11:51.853374 21333 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 21:11:51.857055 21333 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39691
I20260924 21:11:51.857084 21400 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39691 every 8 connection(s)
I20260924 21:11:51.857372 21333 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790284076800039-392-0/minicluster-data/master-0/data/info.pb
I20260924 21:11:51.858134 21401 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 21:11:51.859956 21401 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c50343ab2c924f66ab22b75f46f118ac: Bootstrap starting.
I20260924 21:11:51.860426 21401 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c50343ab2c924f66ab22b75f46f118ac: Neither blocks nor log segments found. Creating new log.
I20260924 21:11:51.860644 21401 log.cc:824] T 00000000000000000000000000000000 P c50343ab2c924f66ab22b75f46f118ac: Log is configured to *not* fsync() on all Append() calls
I20260924 21:11:51.861254 21401 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c50343ab2c924f66ab22b75f46f118ac: No bootstrap required, opened a new log
I20260924 21:11:51.861682 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAzNNjS/build/release/bin/kudu as pid 21333
I20260924 21:11:51.861847 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790284076800039-392-0/minicluster-data/master-0/wal/instance
I20260924 21:11:51.862025 392 external_mini_cluster.cc:1428] Setting key 858930f4fb0f92f54b2bd09515fb3b28
I20260924 21:11:51.863282 21401 raft_consensus.cc:348] T 00000000000000000000000000000000 P c50343ab2c924f66ab22b75f46f118ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c50343ab2c924f66ab22b75f46f118ac" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39691 } }
I20260924 21:11:51.863454 21401 raft_consensus.cc:374] T 00000000000000000000000000000000 P c50343ab2c924f66ab22b75f46f118ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 21:11:51.863485 21401 raft_consensus.cc:729] T 00000000000000000000000000000000 P c50343ab2c924f66ab22b75f46f118ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c50343ab2c924f66ab22b75f46f118ac, State: Initialized, Role: FOLLOWER
I20260924 21:11:51.863596 21401 consensus_queue.cc:260] T 00000000000000000000000000000000 P c50343ab2c924f66ab22b75f46f118ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c50343ab2c924f66ab22b75f46f118ac" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39691 } }
I20260924 21:11:51.863674 21401 raft_consensus.cc:388] T 00000000000000000000000000000000 P c50343ab2c924f66ab22b75f46f118ac [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 21:11:51.863703 21401 raft_consensus.cc:482] T 00000000000000000000000000000000 P c50343ab2c924f66ab22b75f46f118ac [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 21:11:51.863787 21401 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c50343ab2c924f66ab22b75f46f118ac [term 0 FOLLOWER]: Advancing to term 1
I20260924 21:11:51.864337 21401 raft_consensus.cc:504] T 00000000000000000000000000000000 P c50343ab2c924f66ab22b75f46f118ac [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c50343ab2c924f66ab22b75f46f118ac" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39691 } }
I20260924 21:11:51.864425 21401 leader_election.cc:302] T 00000000000000000000000000000000 P c50343ab2c924f66ab22b75f46f118ac [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c50343ab2c924f66ab22b75f46f118ac; no voters:
I20260924 21:11:51.864559 21401 leader_election.cc:288] T 00000000000000000000000000000000 P c50343ab2c924f66ab22b75f46f118ac [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 21:11:51.864609 21406 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c50343ab2c924f66ab22b75f46f118ac [term 1 FOLLOWER]: Leader election won for term 1
I20260924 21:11:51.864759 21401 sys_catalog.cc:564] T 00000000000000000000000000000000 P c50343ab2c924f66ab22b75f46f118ac [sys.catalog]: configured and running, proceeding with master startup.
I20260924 21:11:51.864846 21406 raft_consensus.cc:686] T 00000000000000000000000000000000 P c50343ab2c924f66ab22b75f46f118ac [term 1 LEADER]: Becoming Leader. State: Replica: c50343ab2c924f66ab22b75f46f118ac, State: Running, Role: LEADER
I20260924 21:11:51.864943 21406 consensus_queue.cc:237] T 00000000000000000000000000000000 P c50343ab2c924f66ab22b75f46f118ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c50343ab2c924f66ab22b75f46f118ac" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39691 } }
I20260924 21:11:51.865252 21406 sys_catalog.cc:455] T 00000000000000000000000000000000 P c50343ab2c924f66ab22b75f46f118ac [sys.catalog]: SysCatalogTable state changed. Reason: New leader c50343ab2c924f66ab22b75f46f118ac. Latest consensus state: current_term: 1 leader_uuid: "c50343ab2c924f66ab22b75f46f118ac" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c50343ab2c924f66ab22b75f46f118ac" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39691 } } }
I20260924 21:11:51.865258 21407 sys_catalog.cc:455] T 00000000000000000000000000000000 P c50343ab2c924f66ab22b75f46f118ac [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c50343ab2c924f66ab22b75f46f118ac" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c50343ab2c924f66ab22b75f46f118ac" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39691 } } }
I20260924 21:11:51.865387 21406 sys_catalog.cc:458] T 00000000000000000000000000000000 P c50343ab2c924f66ab22b75f46f118ac [sys.catalog]: This master's current role is: LEADER
I20260924 21:11:51.865396 21407 sys_catalog.cc:458] T 00000000000000000000000000000000 P c50343ab2c924f66ab22b75f46f118ac [sys.catalog]: This master's current role is: LEADER
I20260924 21:11:51.865680 21415 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 21:11:51.866350 21415 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 21:11:51.867810 21415 catalog_manager.cc:1347] Generated new cluster ID: dfcb5a310d3648269e6be0902d630463
I20260924 21:11:51.867851 21415 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 21:11:51.868285 21405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:11:51.862479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.21:57175 (local address 127.0.0.1:39691)
0924 21:11:51.862681 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0924 21:11:51.862689 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 21:11:51.862717 (+ 28us) server_negotiation.cc:407] Connection header received
0924 21:11:51.863449 (+ 732us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:11:51.863457 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:11:51.863521 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:11:51.863651 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:11:51.863844 (+ 193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:11:51.864988 (+ 1144us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:11:51.866242 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:11:51.866861 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:11:51.866990 (+ 129us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:11:51.867605 (+ 615us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 21:11:51.867608 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 21:11:51.867609 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 21:11:51.867823 (+ 214us) server_negotiation.cc:922] calling sasl_server_start()
0924 21:11:51.867858 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 21:11:51.867868 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 21:11:51.867945 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0924 21:11:51.868143 (+ 198us) server_negotiation.cc:299] Negotiation successful
0924 21:11:51.868161 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":97,"threads_started":1}
I20260924 21:11:51.869122 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 21:11:51.873848 21415 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 21:11:51.874430 21415 catalog_manager.cc:1504] Loading token signing keys...
I20260924 21:11:51.876008 21405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:11:51.870436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.21:40469 (local address 127.0.0.1:39691)
0924 21:11:51.870469 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0924 21:11:51.870470 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 21:11:51.873641 (+ 3171us) server_negotiation.cc:407] Connection header received
0924 21:11:51.873663 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:11:51.873664 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:11:51.873673 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:11:51.873728 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:11:51.873964 (+ 236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:11:51.874363 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:11:51.875020 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:11:51.875200 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:11:51.875262 (+ 62us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:11:51.875791 (+ 529us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 21:11:51.875793 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 21:11:51.875793 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 21:11:51.875797 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 21:11:51.875807 (+ 10us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 21:11:51.875811 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 21:11:51.875859 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0924 21:11:51.875913 (+ 54us) server_negotiation.cc:299] Negotiation successful
0924 21:11:51.875920 (+ 7us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260924 21:11:51.882597 21415 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c50343ab2c924f66ab22b75f46f118ac: Generated new TSK 0
I20260924 21:11:51.882740 21415 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 21:11:51.884384 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAzNNjS/build/release/bin/kudu with pid 21333
W20260924 21:11:51.888443 21318 connection.cc:535] client connection to 127.0.0.1:39691 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:39691): Connection reset by peer (error 104)
2026-09-24T21:11:51Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1149 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790284076800039-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 21:11:51 dist-test-slave-c7tk krb5kdc[21432](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 21:11:51 dist-test-slave-c7tk krb5kdc[21432](info): set up 2 sockets
Sep 24 21:11:51 dist-test-slave-c7tk krb5kdc[21432](info): commencing operation
krb5kdc: starting...
W20260924 21:11:52.801695 392 mini_kdc.cc:121] Time spent starting KDC: real 0.900s user 0.000s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790284076800039-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790284076800039-392-0/krb5kdc/test-user.keytab.
Sep 24 21:11:52 dist-test-slave-c7tk krb5kdc[21432](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284312, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T21:11:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T21:11:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790284076800039-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790284076800039-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 21:11:52.902266 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAzNNjS/build/release/bin/kudu
/tmp/dist-test-taskAzNNjS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790284076800039-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790284076800039-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790284076800039-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790284076800039-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44889
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790284076800039-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36515
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44889
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790284076800039-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790284076800039-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790284076800039-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 21:11:52.970810 21445 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 21:11:52.970960 21445 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 21:11:52.970978 21445 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 21:11:52.971895 21445 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 21:11:52.971923 21445 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 21:11:52.971935 21445 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 21:11:52.971946 21445 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 21:11:52.971956 21445 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 21:11:52.973124 21445 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36515
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790284076800039-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790284076800039-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44889
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790284076800039-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44889
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790284076800039-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21445
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790284076800039-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 19:49:06 UTC on ccycloud.quasar-vhqjoc.root.comops.site
build id 4670864
I20260924 21:11:52.973269 21445 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 21:11:52.973397 21445 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 21:11:52.977485 21451 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 21:11:52.977581 21445 server_base.cc:1034] running on GCE node
W20260924 21:11:52.977545 21450 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 21:11:52.977501 21453 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 21:11:52.977900 21445 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 21:11:52.978214 21445 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 21:11:52.979393 21445 hybrid_clock.cc:648] HybridClock initialized: now 1790284312979362 us; error 32 us; skew 500 ppm
Sep 24 21:11:52 dist-test-slave-c7tk krb5kdc[21432](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284312, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 21:11:52.981544 21445 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 21:11:52.981980 21445 webserver.cc:460] Webserver started at http://127.0.0.1:45957/ using document root <none> and password file <none>
I20260924 21:11:52.982182 21445 fs_manager.cc:359] Metadata directory not provided
I20260924 21:11:52.982237 21445 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 21:11:52.982345 21445 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 21:11:52.983165 21445 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790284076800039-392-0/minicluster-data/master-0/data/instance:
uuid: "0cf4acc3b627491f8471fdff281bf770"
format_stamp: "Formatted at 2026-09-24 21:11:52 on dist-test-slave-c7tk"
server_key: "35710f9370dead5d21ba8a06e406003e"
server_key_iv: "7a3fbd2c2bad5f9c371e6b15b82a7be4"
server_key_version: "encryptionkey@0"
I20260924 21:11:52.983456 21445 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790284076800039-392-0/minicluster-data/master-0/wal/instance:
uuid: "0cf4acc3b627491f8471fdff281bf770"
format_stamp: "Formatted at 2026-09-24 21:11:52 on dist-test-slave-c7tk"
server_key: "35710f9370dead5d21ba8a06e406003e"
server_key_iv: "7a3fbd2c2bad5f9c371e6b15b82a7be4"
server_key_version: "encryptionkey@0"
I20260924 21:11:52.984615 21445 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 21:11:52.985205 21460 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 21:11:52.985355 21445 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 21:11:52.985435 21445 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790284076800039-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790284076800039-392-0/minicluster-data/master-0/wal
uuid: "0cf4acc3b627491f8471fdff281bf770"
format_stamp: "Formatted at 2026-09-24 21:11:52 on dist-test-slave-c7tk"
server_key: "35710f9370dead5d21ba8a06e406003e"
server_key_iv: "7a3fbd2c2bad5f9c371e6b15b82a7be4"
server_key_version: "encryptionkey@0"
I20260924 21:11:52.985507 21445 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790284076800039-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790284076800039-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790284076800039-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 21:11:52.988168 21445 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 21:11:52.989615 21445 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 21:11:52.989763 21445 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 21:11:52.993292 21445 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44889
I20260924 21:11:52.993325 21512 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44889 every 8 connection(s)
I20260924 21:11:52.993614 21445 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790284076800039-392-0/minicluster-data/master-0/data/info.pb
I20260924 21:11:52.994513 21513 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 21:11:52.995407 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAzNNjS/build/release/bin/kudu as pid 21445
I20260924 21:11:52.995524 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790284076800039-392-0/minicluster-data/master-0/wal/instance
I20260924 21:11:52.995702 392 external_mini_cluster.cc:1428] Setting key 1f5b25b95af487770b90a02cce2c2a14
I20260924 21:11:52.996727 21513 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0cf4acc3b627491f8471fdff281bf770: Bootstrap starting.
I20260924 21:11:52.997354 21513 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0cf4acc3b627491f8471fdff281bf770: Neither blocks nor log segments found. Creating new log.
I20260924 21:11:52.997644 21513 log.cc:824] T 00000000000000000000000000000000 P 0cf4acc3b627491f8471fdff281bf770: Log is configured to *not* fsync() on all Append() calls
I20260924 21:11:52.997699 21516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:11:52.996133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.21:36393 (local address 127.0.0.1:44889)
0924 21:11:52.996294 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0924 21:11:52.996300 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 21:11:52.996330 (+ 30us) server_negotiation.cc:407] Connection header received
0924 21:11:52.996654 (+ 324us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:11:52.996659 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:11:52.996677 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:11:52.996770 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:11:52.997302 (+ 532us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 21:11:52.997304 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 21:11:52.997305 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 21:11:52.997307 (+ 2us) server_negotiation.cc:922] calling sasl_server_start()
0924 21:11:52.997335 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 21:11:52.997343 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 21:11:52.997400 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0924 21:11:52.997488 (+ 88us) server_negotiation.cc:299] Negotiation successful
0924 21:11:52.997526 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":96,"threads_started":1}
I20260924 21:11:52.998282 21513 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0cf4acc3b627491f8471fdff281bf770: No bootstrap required, opened a new log
I20260924 21:11:52.998569 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 21:11:53.000643 21513 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0cf4acc3b627491f8471fdff281bf770 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0cf4acc3b627491f8471fdff281bf770" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44889 } }
I20260924 21:11:53.000860 21513 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0cf4acc3b627491f8471fdff281bf770 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 21:11:53.000907 21513 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0cf4acc3b627491f8471fdff281bf770 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cf4acc3b627491f8471fdff281bf770, State: Initialized, Role: FOLLOWER
I20260924 21:11:53.000993 21516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:11:52.999919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.21:55611 (local address 127.0.0.1:44889)
0924 21:11:52.999953 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0924 21:11:52.999955 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 21:11:53.000041 (+ 86us) server_negotiation.cc:407] Connection header received
0924 21:11:53.000123 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:11:53.000124 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:11:53.000127 (+ 3us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:11:53.000168 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:11:53.000676 (+ 508us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 21:11:53.000676 (+ 0us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 21:11:53.000677 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 21:11:53.000680 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0924 21:11:53.000692 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 21:11:53.000696 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 21:11:53.000748 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0924 21:11:53.000878 (+ 130us) server_negotiation.cc:299] Negotiation successful
0924 21:11:53.000888 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260924 21:11:53.001031 21513 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0cf4acc3b627491f8471fdff281bf770 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0cf4acc3b627491f8471fdff281bf770" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44889 } }
I20260924 21:11:53.001119 21513 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0cf4acc3b627491f8471fdff281bf770 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 21:11:53.001151 21513 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0cf4acc3b627491f8471fdff281bf770 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 21:11:53.001194 21513 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0cf4acc3b627491f8471fdff281bf770 [term 0 FOLLOWER]: Advancing to term 1
I20260924 21:11:53.001616 21513 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0cf4acc3b627491f8471fdff281bf770 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0cf4acc3b627491f8471fdff281bf770" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44889 } }
I20260924 21:11:53.001686 21513 leader_election.cc:302] T 00000000000000000000000000000000 P 0cf4acc3b627491f8471fdff281bf770 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0cf4acc3b627491f8471fdff281bf770; no voters:
I20260924 21:11:53.001850 21513 leader_election.cc:288] T 00000000000000000000000000000000 P 0cf4acc3b627491f8471fdff281bf770 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 21:11:53.001878 21523 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0cf4acc3b627491f8471fdff281bf770 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 21:11:53.001999 21523 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0cf4acc3b627491f8471fdff281bf770 [term 1 LEADER]: Becoming Leader. State: Replica: 0cf4acc3b627491f8471fdff281bf770, State: Running, Role: LEADER
I20260924 21:11:53.002064 21513 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0cf4acc3b627491f8471fdff281bf770 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 21:11:53.002104 21523 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0cf4acc3b627491f8471fdff281bf770 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0cf4acc3b627491f8471fdff281bf770" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44889 } }
I20260924 21:11:53.002447 21524 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0cf4acc3b627491f8471fdff281bf770 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0cf4acc3b627491f8471fdff281bf770" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0cf4acc3b627491f8471fdff281bf770" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44889 } } }
I20260924 21:11:53.002449 21525 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0cf4acc3b627491f8471fdff281bf770 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0cf4acc3b627491f8471fdff281bf770. Latest consensus state: current_term: 1 leader_uuid: "0cf4acc3b627491f8471fdff281bf770" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0cf4acc3b627491f8471fdff281bf770" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44889 } } }
I20260924 21:11:53.002592 21524 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0cf4acc3b627491f8471fdff281bf770 [sys.catalog]: This master's current role is: LEADER
I20260924 21:11:53.002743 21525 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0cf4acc3b627491f8471fdff281bf770 [sys.catalog]: This master's current role is: LEADER
I20260924 21:11:53.002925 21531 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 21:11:53.003319 21531 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 21:11:53.004582 21531 catalog_manager.cc:1347] Generated new cluster ID: bbcb817dabf84554a65ba2711daefa6a
I20260924 21:11:53.004630 21531 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 21:11:53.010672 21531 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 21:11:53.010825 21531 catalog_manager.cc:1504] Loading token signing keys...
I20260924 21:11:53.020882 21531 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0cf4acc3b627491f8471fdff281bf770: Generated new TSK 0
I20260924 21:11:53.021036 21531 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 21:11:53.025585 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAzNNjS/build/release/bin/kudu with pid 21445
2026-09-24T21:11:53Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1141 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790284076800039-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 21:11:53 dist-test-slave-c7tk krb5kdc[21544](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 21:11:53 dist-test-slave-c7tk krb5kdc[21544](info): set up 2 sockets
Sep 24 21:11:53 dist-test-slave-c7tk krb5kdc[21544](info): commencing operation
krb5kdc: starting...
W20260924 21:11:53.965271 392 mini_kdc.cc:121] Time spent starting KDC: real 0.922s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790284076800039-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790284076800039-392-0/krb5kdc/test-user.keytab.
Sep 24 21:11:54 dist-test-slave-c7tk krb5kdc[21544](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284314, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T21:11:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T21:11:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790284076800039-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790284076800039-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 21:11:54.068435 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAzNNjS/build/release/bin/kudu
/tmp/dist-test-taskAzNNjS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790284076800039-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790284076800039-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790284076800039-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790284076800039-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37941
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790284076800039-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34489
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37941
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790284076800039-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790284076800039-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790284076800039-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 21:11:54.138326 21557 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 21:11:54.138525 21557 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 21:11:54.138563 21557 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 21:11:54.139604 21557 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 21:11:54.139667 21557 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 21:11:54.139689 21557 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 21:11:54.139734 21557 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 21:11:54.139757 21557 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 21:11:54.141170 21557 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34489
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790284076800039-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790284076800039-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37941
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790284076800039-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37941
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790284076800039-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21557
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790284076800039-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 19:49:06 UTC on ccycloud.quasar-vhqjoc.root.comops.site
build id 4670864
I20260924 21:11:54.141398 21557 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 21:11:54.141603 21557 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 21:11:54.145640 21562 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 21:11:54.145695 21563 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 21:11:54.145725 21557 server_base.cc:1034] running on GCE node
W20260924 21:11:54.145663 21565 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 21:11:54.146082 21557 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 21:11:54.146360 21557 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 21:11:54.147512 21557 hybrid_clock.cc:648] HybridClock initialized: now 1790284314147492 us; error 26 us; skew 500 ppm
Sep 24 21:11:54 dist-test-slave-c7tk krb5kdc[21544](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284314, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 21:11:54.149518 21557 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 21:11:54.149966 21557 webserver.cc:460] Webserver started at http://127.0.0.1:37351/ using document root <none> and password file <none>
I20260924 21:11:54.150195 21557 fs_manager.cc:359] Metadata directory not provided
I20260924 21:11:54.150261 21557 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 21:11:54.150405 21557 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 21:11:54.151297 21557 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790284076800039-392-0/minicluster-data/master-0/data/instance:
uuid: "5cd12b7c5d98445aaaeeef1ad8f7dc6e"
format_stamp: "Formatted at 2026-09-24 21:11:54 on dist-test-slave-c7tk"
server_key: "d79eee8a0475b787f81684b9f0209bae"
server_key_iv: "24a83b93dc0c666f8397faa8a6653c48"
server_key_version: "encryptionkey@0"
I20260924 21:11:54.151618 21557 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790284076800039-392-0/minicluster-data/master-0/wal/instance:
uuid: "5cd12b7c5d98445aaaeeef1ad8f7dc6e"
format_stamp: "Formatted at 2026-09-24 21:11:54 on dist-test-slave-c7tk"
server_key: "d79eee8a0475b787f81684b9f0209bae"
server_key_iv: "24a83b93dc0c666f8397faa8a6653c48"
server_key_version: "encryptionkey@0"
I20260924 21:11:54.152741 21557 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 21:11:54.153312 21572 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 21:11:54.153445 21557 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 21:11:54.153548 21557 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790284076800039-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790284076800039-392-0/minicluster-data/master-0/wal
uuid: "5cd12b7c5d98445aaaeeef1ad8f7dc6e"
format_stamp: "Formatted at 2026-09-24 21:11:54 on dist-test-slave-c7tk"
server_key: "d79eee8a0475b787f81684b9f0209bae"
server_key_iv: "24a83b93dc0c666f8397faa8a6653c48"
server_key_version: "encryptionkey@0"
I20260924 21:11:54.153620 21557 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790284076800039-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790284076800039-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790284076800039-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 21:11:54.163450 21557 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 21:11:54.164896 21557 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 21:11:54.165040 21557 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 21:11:54.168447 21557 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37941
I20260924 21:11:54.168493 21624 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37941 every 8 connection(s)
I20260924 21:11:54.168859 21557 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790284076800039-392-0/minicluster-data/master-0/data/info.pb
I20260924 21:11:54.169574 21625 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 21:11:54.171557 21625 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5cd12b7c5d98445aaaeeef1ad8f7dc6e: Bootstrap starting.
I20260924 21:11:54.172045 21625 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5cd12b7c5d98445aaaeeef1ad8f7dc6e: Neither blocks nor log segments found. Creating new log.
I20260924 21:11:54.172053 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAzNNjS/build/release/bin/kudu as pid 21557
I20260924 21:11:54.172151 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790284076800039-392-0/minicluster-data/master-0/wal/instance
I20260924 21:11:54.172288 392 external_mini_cluster.cc:1428] Setting key fdb4c4a02e5f9dadd23cae93da0ab184
I20260924 21:11:54.172314 21625 log.cc:824] T 00000000000000000000000000000000 P 5cd12b7c5d98445aaaeeef1ad8f7dc6e: Log is configured to *not* fsync() on all Append() calls
I20260924 21:11:54.172973 21625 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5cd12b7c5d98445aaaeeef1ad8f7dc6e: No bootstrap required, opened a new log
I20260924 21:11:54.174667 21625 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5cd12b7c5d98445aaaeeef1ad8f7dc6e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cd12b7c5d98445aaaeeef1ad8f7dc6e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37941 } }
I20260924 21:11:54.174795 21625 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5cd12b7c5d98445aaaeeef1ad8f7dc6e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 21:11:54.174849 21625 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5cd12b7c5d98445aaaeeef1ad8f7dc6e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5cd12b7c5d98445aaaeeef1ad8f7dc6e, State: Initialized, Role: FOLLOWER
I20260924 21:11:54.174976 21625 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5cd12b7c5d98445aaaeeef1ad8f7dc6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cd12b7c5d98445aaaeeef1ad8f7dc6e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37941 } }
I20260924 21:11:54.175064 21625 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5cd12b7c5d98445aaaeeef1ad8f7dc6e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 21:11:54.175102 21625 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5cd12b7c5d98445aaaeeef1ad8f7dc6e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 21:11:54.175143 21625 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5cd12b7c5d98445aaaeeef1ad8f7dc6e [term 0 FOLLOWER]: Advancing to term 1
I20260924 21:11:54.175560 21625 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5cd12b7c5d98445aaaeeef1ad8f7dc6e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cd12b7c5d98445aaaeeef1ad8f7dc6e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37941 } }
I20260924 21:11:54.175663 21625 leader_election.cc:302] T 00000000000000000000000000000000 P 5cd12b7c5d98445aaaeeef1ad8f7dc6e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5cd12b7c5d98445aaaeeef1ad8f7dc6e; no voters:
I20260924 21:11:54.175808 21625 leader_election.cc:288] T 00000000000000000000000000000000 P 5cd12b7c5d98445aaaeeef1ad8f7dc6e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 21:11:54.175841 21630 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5cd12b7c5d98445aaaeeef1ad8f7dc6e [term 1 FOLLOWER]: Leader election won for term 1
I20260924 21:11:54.175938 21630 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5cd12b7c5d98445aaaeeef1ad8f7dc6e [term 1 LEADER]: Becoming Leader. State: Replica: 5cd12b7c5d98445aaaeeef1ad8f7dc6e, State: Running, Role: LEADER
I20260924 21:11:54.176033 21630 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5cd12b7c5d98445aaaeeef1ad8f7dc6e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cd12b7c5d98445aaaeeef1ad8f7dc6e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37941 } }
I20260924 21:11:54.176034 21625 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5cd12b7c5d98445aaaeeef1ad8f7dc6e [sys.catalog]: configured and running, proceeding with master startup.
I20260924 21:11:54.176630 21631 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5cd12b7c5d98445aaaeeef1ad8f7dc6e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5cd12b7c5d98445aaaeeef1ad8f7dc6e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cd12b7c5d98445aaaeeef1ad8f7dc6e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37941 } } }
I20260924 21:11:54.176695 21631 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5cd12b7c5d98445aaaeeef1ad8f7dc6e [sys.catalog]: This master's current role is: LEADER
I20260924 21:11:54.176834 21632 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5cd12b7c5d98445aaaeeef1ad8f7dc6e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5cd12b7c5d98445aaaeeef1ad8f7dc6e. Latest consensus state: current_term: 1 leader_uuid: "5cd12b7c5d98445aaaeeef1ad8f7dc6e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cd12b7c5d98445aaaeeef1ad8f7dc6e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37941 } } }
I20260924 21:11:54.176851 21639 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 21:11:54.176888 21632 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5cd12b7c5d98445aaaeeef1ad8f7dc6e [sys.catalog]: This master's current role is: LEADER
I20260924 21:11:54.177248 21639 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 24 21:11:54 dist-test-slave-c7tk krb5kdc[21544](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284314, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260924 21:11:54.181543 21629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:11:54.172655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.21:58801 (local address 127.0.0.1:37941)
0924 21:11:54.172819 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0924 21:11:54.172827 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 21:11:54.172853 (+ 26us) server_negotiation.cc:407] Connection header received
0924 21:11:54.173247 (+ 394us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:11:54.173254 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:11:54.173295 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:11:54.173416 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:11:54.174048 (+ 632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:11:54.174742 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:11:54.175551 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:11:54.176035 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:11:54.176390 (+ 355us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:11:54.178634 (+ 2244us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 21:11:54.178636 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 21:11:54.178637 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 21:11:54.178644 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 21:11:54.180068 (+ 1424us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:11:54.180492 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:11:54.180495 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:11:54.180495 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:11:54.180566 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:11:54.180788 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:11:54.180789 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:11:54.180789 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:11:54.180994 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 21:11:54.181115 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0924 21:11:54.181257 (+ 142us) server_negotiation.cc:299] Negotiation successful
0924 21:11:54.181318 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116,"thread_start_us":94,"threads_started":1}
I20260924 21:11:54.182226 21639 catalog_manager.cc:1347] Generated new cluster ID: 09a499c1aa7e4a2c91b7f3ee133b22c7
I20260924 21:11:54.182276 21639 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 21:11:54.182390 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 21:11:54.189725 21629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:11:54.183787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.21:49541 (local address 127.0.0.1:37941)
0924 21:11:54.183815 (+ 28us) server_negotiation.cc:206] Beginning negotiation
0924 21:11:54.183816 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 21:11:54.183971 (+ 155us) server_negotiation.cc:407] Connection header received
0924 21:11:54.184017 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:11:54.184018 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:11:54.184025 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:11:54.184072 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:11:54.184585 (+ 513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:11:54.185163 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:11:54.186020 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:11:54.186284 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:11:54.186333 (+ 49us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:11:54.187457 (+ 1124us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 21:11:54.187458 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 21:11:54.187459 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 21:11:54.187471 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 21:11:54.188691 (+ 1220us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:11:54.189038 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:11:54.189039 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:11:54.189039 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:11:54.189084 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:11:54.189273 (+ 189us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:11:54.189273 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:11:54.189273 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:11:54.189361 (+ 88us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 21:11:54.189454 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0924 21:11:54.189563 (+ 109us) server_negotiation.cc:299] Negotiation successful
0924 21:11:54.189640 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260924 21:11:54.190852 21639 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 21:11:54.191428 21639 catalog_manager.cc:1504] Loading token signing keys...
I20260924 21:11:54.195555 21639 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5cd12b7c5d98445aaaeeef1ad8f7dc6e: Generated new TSK 0
I20260924 21:11:54.195683 21639 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 21:11:54.199218 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAzNNjS/build/release/bin/kudu with pid 21557
W20260924 21:11:54.202818 21542 connection.cc:535] client connection to 127.0.0.1:37941 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:37941): Connection reset by peer (error 104)
2026-09-24T21:11:54Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1172 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790284076800039-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 21:11:54 dist-test-slave-c7tk krb5kdc[21656](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 21:11:54 dist-test-slave-c7tk krb5kdc[21656](info): set up 2 sockets
Sep 24 21:11:54 dist-test-slave-c7tk krb5kdc[21656](info): commencing operation
krb5kdc: starting...
W20260924 21:11:55.133615 392 mini_kdc.cc:121] Time spent starting KDC: real 0.918s user 0.000s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790284076800039-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790284076800039-392-0/krb5kdc/test-user.keytab.
Sep 24 21:11:55 dist-test-slave-c7tk krb5kdc[21656](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284315, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T21:11:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T21:11:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790284076800039-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790284076800039-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 21:11:55.234838 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAzNNjS/build/release/bin/kudu
/tmp/dist-test-taskAzNNjS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790284076800039-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790284076800039-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790284076800039-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790284076800039-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45787
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790284076800039-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45397
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45787
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790284076800039-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790284076800039-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790284076800039-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 21:11:55.305434 21669 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 21:11:55.305653 21669 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 21:11:55.305692 21669 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 21:11:55.306799 21669 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 21:11:55.306862 21669 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 21:11:55.306885 21669 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 21:11:55.306906 21669 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 21:11:55.306947 21669 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 21:11:55.308394 21669 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45397
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790284076800039-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790284076800039-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45787
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790284076800039-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45787
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790284076800039-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21669
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790284076800039-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 19:49:06 UTC on ccycloud.quasar-vhqjoc.root.comops.site
build id 4670864
I20260924 21:11:55.308627 21669 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 21:11:55.308815 21669 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 21:11:55.313316 21674 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 21:11:55.313351 21677 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 21:11:55.313326 21675 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 21:11:55.313630 21669 server_base.cc:1034] running on GCE node
I20260924 21:11:55.313917 21669 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 21:11:55.314261 21669 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 21:11:55.315398 21669 hybrid_clock.cc:648] HybridClock initialized: now 1790284315315385 us; error 27 us; skew 500 ppm
Sep 24 21:11:55 dist-test-slave-c7tk krb5kdc[21656](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284315, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 21:11:55.317588 21669 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 21:11:55.318019 21669 webserver.cc:460] Webserver started at http://127.0.0.1:33299/ using document root <none> and password file <none>
I20260924 21:11:55.318222 21669 fs_manager.cc:359] Metadata directory not provided
I20260924 21:11:55.318262 21669 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 21:11:55.318385 21669 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 21:11:55.319315 21669 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790284076800039-392-0/minicluster-data/master-0/data/instance:
uuid: "c793f8364ce94df0aff3459cb887d6cb"
format_stamp: "Formatted at 2026-09-24 21:11:55 on dist-test-slave-c7tk"
server_key: "ff30f90cea405a3010486547a6145f8f"
server_key_iv: "6fa35bd9c3645fdd1097dde99c89186b"
server_key_version: "encryptionkey@0"
I20260924 21:11:55.319628 21669 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790284076800039-392-0/minicluster-data/master-0/wal/instance:
uuid: "c793f8364ce94df0aff3459cb887d6cb"
format_stamp: "Formatted at 2026-09-24 21:11:55 on dist-test-slave-c7tk"
server_key: "ff30f90cea405a3010486547a6145f8f"
server_key_iv: "6fa35bd9c3645fdd1097dde99c89186b"
server_key_version: "encryptionkey@0"
I20260924 21:11:55.320858 21669 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 21:11:55.321595 21684 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 21:11:55.321767 21669 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 21:11:55.321893 21669 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790284076800039-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790284076800039-392-0/minicluster-data/master-0/wal
uuid: "c793f8364ce94df0aff3459cb887d6cb"
format_stamp: "Formatted at 2026-09-24 21:11:55 on dist-test-slave-c7tk"
server_key: "ff30f90cea405a3010486547a6145f8f"
server_key_iv: "6fa35bd9c3645fdd1097dde99c89186b"
server_key_version: "encryptionkey@0"
I20260924 21:11:55.321978 21669 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790284076800039-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790284076800039-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790284076800039-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 21:11:55.336351 21669 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 21:11:55.338167 21669 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 21:11:55.338315 21669 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 21:11:55.342065 21669 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45787
I20260924 21:11:55.342097 21736 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45787 every 8 connection(s)
I20260924 21:11:55.342397 21669 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790284076800039-392-0/minicluster-data/master-0/data/info.pb
I20260924 21:11:55.343247 21737 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 21:11:55.345053 21737 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c793f8364ce94df0aff3459cb887d6cb: Bootstrap starting.
I20260924 21:11:55.345489 21737 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c793f8364ce94df0aff3459cb887d6cb: Neither blocks nor log segments found. Creating new log.
I20260924 21:11:55.345691 21737 log.cc:824] T 00000000000000000000000000000000 P c793f8364ce94df0aff3459cb887d6cb: Log is configured to *not* fsync() on all Append() calls
I20260924 21:11:55.346306 21737 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c793f8364ce94df0aff3459cb887d6cb: No bootstrap required, opened a new log
I20260924 21:11:55.347662 21737 raft_consensus.cc:348] T 00000000000000000000000000000000 P c793f8364ce94df0aff3459cb887d6cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c793f8364ce94df0aff3459cb887d6cb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45787 } }
I20260924 21:11:55.347776 21737 raft_consensus.cc:374] T 00000000000000000000000000000000 P c793f8364ce94df0aff3459cb887d6cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 21:11:55.347796 21737 raft_consensus.cc:729] T 00000000000000000000000000000000 P c793f8364ce94df0aff3459cb887d6cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c793f8364ce94df0aff3459cb887d6cb, State: Initialized, Role: FOLLOWER
I20260924 21:11:55.347913 21737 consensus_queue.cc:260] T 00000000000000000000000000000000 P c793f8364ce94df0aff3459cb887d6cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c793f8364ce94df0aff3459cb887d6cb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45787 } }
I20260924 21:11:55.348012 21737 raft_consensus.cc:388] T 00000000000000000000000000000000 P c793f8364ce94df0aff3459cb887d6cb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 21:11:55.348058 21737 raft_consensus.cc:482] T 00000000000000000000000000000000 P c793f8364ce94df0aff3459cb887d6cb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 21:11:55.348101 21737 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c793f8364ce94df0aff3459cb887d6cb [term 0 FOLLOWER]: Advancing to term 1
I20260924 21:11:55.348148 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAzNNjS/build/release/bin/kudu as pid 21669
I20260924 21:11:55.348251 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790284076800039-392-0/minicluster-data/master-0/wal/instance
I20260924 21:11:55.348434 392 external_mini_cluster.cc:1428] Setting key d51ad326c06a701a3a624f6d8c3e75a5
I20260924 21:11:55.348724 21737 raft_consensus.cc:504] T 00000000000000000000000000000000 P c793f8364ce94df0aff3459cb887d6cb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c793f8364ce94df0aff3459cb887d6cb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45787 } }
I20260924 21:11:55.348816 21737 leader_election.cc:302] T 00000000000000000000000000000000 P c793f8364ce94df0aff3459cb887d6cb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c793f8364ce94df0aff3459cb887d6cb; no voters:
I20260924 21:11:55.349056 21737 leader_election.cc:288] T 00000000000000000000000000000000 P c793f8364ce94df0aff3459cb887d6cb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 21:11:55.349084 21741 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c793f8364ce94df0aff3459cb887d6cb [term 1 FOLLOWER]: Leader election won for term 1
I20260924 21:11:55.349184 21741 raft_consensus.cc:686] T 00000000000000000000000000000000 P c793f8364ce94df0aff3459cb887d6cb [term 1 LEADER]: Becoming Leader. State: Replica: c793f8364ce94df0aff3459cb887d6cb, State: Running, Role: LEADER
I20260924 21:11:55.349284 21737 sys_catalog.cc:564] T 00000000000000000000000000000000 P c793f8364ce94df0aff3459cb887d6cb [sys.catalog]: configured and running, proceeding with master startup.
I20260924 21:11:55.349313 21741 consensus_queue.cc:237] T 00000000000000000000000000000000 P c793f8364ce94df0aff3459cb887d6cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c793f8364ce94df0aff3459cb887d6cb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45787 } }
I20260924 21:11:55.349735 21743 sys_catalog.cc:455] T 00000000000000000000000000000000 P c793f8364ce94df0aff3459cb887d6cb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c793f8364ce94df0aff3459cb887d6cb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c793f8364ce94df0aff3459cb887d6cb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45787 } } }
I20260924 21:11:55.349891 21744 sys_catalog.cc:455] T 00000000000000000000000000000000 P c793f8364ce94df0aff3459cb887d6cb [sys.catalog]: SysCatalogTable state changed. Reason: New leader c793f8364ce94df0aff3459cb887d6cb. Latest consensus state: current_term: 1 leader_uuid: "c793f8364ce94df0aff3459cb887d6cb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c793f8364ce94df0aff3459cb887d6cb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45787 } } }
I20260924 21:11:55.349992 21744 sys_catalog.cc:458] T 00000000000000000000000000000000 P c793f8364ce94df0aff3459cb887d6cb [sys.catalog]: This master's current role is: LEADER
I20260924 21:11:55.349910 21743 sys_catalog.cc:458] T 00000000000000000000000000000000 P c793f8364ce94df0aff3459cb887d6cb [sys.catalog]: This master's current role is: LEADER
I20260924 21:11:55.350301 21751 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 21:11:55.350829 21751 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 21:11:55.352257 21751 catalog_manager.cc:1347] Generated new cluster ID: ac8f551e9da648d384c28bb4828d5cc4
I20260924 21:11:55.352305 21751 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 21:11:55.354143 21742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:11:55.348874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.21:38817 (local address 127.0.0.1:45787)
0924 21:11:55.349105 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0924 21:11:55.349115 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 21:11:55.349143 (+ 28us) server_negotiation.cc:407] Connection header received
0924 21:11:55.349243 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:11:55.349249 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:11:55.349299 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:11:55.349671 (+ 372us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:11:55.349761 (+ 90us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:11:55.351321 (+ 1560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:11:55.352357 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:11:55.352764 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:11:55.352882 (+ 118us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:11:55.353523 (+ 641us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 21:11:55.353527 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 21:11:55.353527 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 21:11:55.353693 (+ 166us) server_negotiation.cc:922] calling sasl_server_start()
0924 21:11:55.353721 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 21:11:55.353730 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 21:11:55.353860 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0924 21:11:55.353948 (+ 88us) server_negotiation.cc:299] Negotiation successful
0924 21:11:55.353975 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":189,"thread_start_us":141,"threads_started":1}
I20260924 21:11:55.354868 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 21:11:55.362231 21751 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 21:11:55.362787 21751 catalog_manager.cc:1504] Loading token signing keys...
I20260924 21:11:55.364324 21742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:11:55.358078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.21:35621 (local address 127.0.0.1:45787)
0924 21:11:55.358114 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0924 21:11:55.358116 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 21:11:55.361991 (+ 3875us) server_negotiation.cc:407] Connection header received
0924 21:11:55.362032 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:11:55.362033 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:11:55.362050 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:11:55.362113 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:11:55.362240 (+ 127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:11:55.362668 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:11:55.363340 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:11:55.363526 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:11:55.363589 (+ 63us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:11:55.364122 (+ 533us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 21:11:55.364123 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 21:11:55.364123 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 21:11:55.364127 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 21:11:55.364138 (+ 11us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 21:11:55.364141 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 21:11:55.364187 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0924 21:11:55.364236 (+ 49us) server_negotiation.cc:299] Negotiation successful
0924 21:11:55.364246 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260924 21:11:55.368942 21751 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c793f8364ce94df0aff3459cb887d6cb: Generated new TSK 0
I20260924 21:11:55.369086 21751 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 21:11:55.372984 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAzNNjS/build/release/bin/kudu with pid 21669
W20260924 21:11:55.376575 21654 connection.cc:535] client connection to 127.0.0.1:45787 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:45787): Connection reset by peer (error 104)
2026-09-24T21:11:55Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1173 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790284076800039-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 21:11:55 dist-test-slave-c7tk krb5kdc[21768](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 21:11:55 dist-test-slave-c7tk krb5kdc[21768](info): set up 2 sockets
Sep 24 21:11:55 dist-test-slave-c7tk krb5kdc[21768](info): commencing operation
krb5kdc: starting...
W20260924 21:11:56.293361 392 mini_kdc.cc:121] Time spent starting KDC: real 0.903s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790284076800039-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790284076800039-392-0/krb5kdc/test-user.keytab.
Sep 24 21:11:56 dist-test-slave-c7tk krb5kdc[21768](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284316, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T21:11:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T21:11:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790284076800039-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790284076800039-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 21:11:56.394788 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAzNNjS/build/release/bin/kudu
/tmp/dist-test-taskAzNNjS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790284076800039-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790284076800039-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790284076800039-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790284076800039-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33525
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790284076800039-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46431
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33525
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790284076800039-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790284076800039-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790284076800039-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 21:11:56.464951 21781 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 21:11:56.465139 21781 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 21:11:56.465174 21781 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 21:11:56.466152 21781 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 21:11:56.466207 21781 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 21:11:56.466228 21781 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 21:11:56.466259 21781 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 21:11:56.466310 21781 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 21:11:56.467702 21781 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46431
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790284076800039-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790284076800039-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33525
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790284076800039-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33525
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790284076800039-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21781
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790284076800039-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 19:49:06 UTC on ccycloud.quasar-vhqjoc.root.comops.site
build id 4670864
I20260924 21:11:56.467938 21781 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 21:11:56.468144 21781 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 21:11:56.472607 21789 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 21:11:56.472604 21787 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 21:11:56.472601 21786 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 21:11:56.472870 21781 server_base.cc:1034] running on GCE node
I20260924 21:11:56.473111 21781 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 21:11:56.473387 21781 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 21:11:56.474542 21781 hybrid_clock.cc:648] HybridClock initialized: now 1790284316474525 us; error 46 us; skew 500 ppm
Sep 24 21:11:56 dist-test-slave-c7tk krb5kdc[21768](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284316, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 21:11:56.481424 21781 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 21:11:56.481909 21781 webserver.cc:460] Webserver started at http://127.0.0.1:36741/ using document root <none> and password file <none>
I20260924 21:11:56.482133 21781 fs_manager.cc:359] Metadata directory not provided
I20260924 21:11:56.482177 21781 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 21:11:56.482352 21781 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 21:11:56.483390 21781 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790284076800039-392-0/minicluster-data/master-0/data/instance:
uuid: "b9f5583bbcfb489e9ad752d58c1b1130"
format_stamp: "Formatted at 2026-09-24 21:11:56 on dist-test-slave-c7tk"
server_key: "1a0add8fc0a44b96e7aaa265ceeb700e"
server_key_iv: "778ddd5167130a6da290b90ecfc1b7f3"
server_key_version: "encryptionkey@0"
I20260924 21:11:56.483728 21781 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790284076800039-392-0/minicluster-data/master-0/wal/instance:
uuid: "b9f5583bbcfb489e9ad752d58c1b1130"
format_stamp: "Formatted at 2026-09-24 21:11:56 on dist-test-slave-c7tk"
server_key: "1a0add8fc0a44b96e7aaa265ceeb700e"
server_key_iv: "778ddd5167130a6da290b90ecfc1b7f3"
server_key_version: "encryptionkey@0"
I20260924 21:11:56.484967 21781 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260924 21:11:56.485641 21796 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 21:11:56.485824 21781 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 21:11:56.485947 21781 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790284076800039-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790284076800039-392-0/minicluster-data/master-0/wal
uuid: "b9f5583bbcfb489e9ad752d58c1b1130"
format_stamp: "Formatted at 2026-09-24 21:11:56 on dist-test-slave-c7tk"
server_key: "1a0add8fc0a44b96e7aaa265ceeb700e"
server_key_iv: "778ddd5167130a6da290b90ecfc1b7f3"
server_key_version: "encryptionkey@0"
I20260924 21:11:56.486048 21781 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790284076800039-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790284076800039-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790284076800039-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 21:11:56.489290 21781 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 21:11:56.491513 21781 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 21:11:56.491653 21781 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 21:11:56.495402 21781 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33525
I20260924 21:11:56.495428 21848 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33525 every 8 connection(s)
I20260924 21:11:56.495711 21781 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790284076800039-392-0/minicluster-data/master-0/data/info.pb
I20260924 21:11:56.496592 21849 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 21:11:56.498538 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAzNNjS/build/release/bin/kudu as pid 21781
I20260924 21:11:56.498625 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790284076800039-392-0/minicluster-data/master-0/wal/instance
I20260924 21:11:56.498713 21849 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b9f5583bbcfb489e9ad752d58c1b1130: Bootstrap starting.
I20260924 21:11:56.498842 392 external_mini_cluster.cc:1428] Setting key 3020f7a5ea8e61bccd80884fe4c15a24
I20260924 21:11:56.499423 21849 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b9f5583bbcfb489e9ad752d58c1b1130: Neither blocks nor log segments found. Creating new log.
I20260924 21:11:56.499720 21849 log.cc:824] T 00000000000000000000000000000000 P b9f5583bbcfb489e9ad752d58c1b1130: Log is configured to *not* fsync() on all Append() calls
I20260924 21:11:56.500430 21849 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b9f5583bbcfb489e9ad752d58c1b1130: No bootstrap required, opened a new log
I20260924 21:11:56.501292 21852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:11:56.499302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.21:44123 (local address 127.0.0.1:33525)
0924 21:11:56.499488 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0924 21:11:56.499496 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 21:11:56.499529 (+ 33us) server_negotiation.cc:407] Connection header received
0924 21:11:56.500093 (+ 564us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:11:56.500122 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:11:56.500155 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:11:56.500299 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:11:56.500878 (+ 579us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 21:11:56.500883 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 21:11:56.500883 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 21:11:56.500887 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 21:11:56.500930 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 21:11:56.500943 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 21:11:56.501035 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0924 21:11:56.501064 (+ 29us) server_negotiation.cc:299] Negotiation successful
0924 21:11:56.501116 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119,"thread_start_us":73,"threads_started":1}
I20260924 21:11:56.502252 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 21:11:56.502671 21849 raft_consensus.cc:348] T 00000000000000000000000000000000 P b9f5583bbcfb489e9ad752d58c1b1130 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9f5583bbcfb489e9ad752d58c1b1130" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33525 } }
I20260924 21:11:56.502781 21849 raft_consensus.cc:374] T 00000000000000000000000000000000 P b9f5583bbcfb489e9ad752d58c1b1130 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 21:11:56.502801 21849 raft_consensus.cc:729] T 00000000000000000000000000000000 P b9f5583bbcfb489e9ad752d58c1b1130 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9f5583bbcfb489e9ad752d58c1b1130, State: Initialized, Role: FOLLOWER
I20260924 21:11:56.502949 21849 consensus_queue.cc:260] T 00000000000000000000000000000000 P b9f5583bbcfb489e9ad752d58c1b1130 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9f5583bbcfb489e9ad752d58c1b1130" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33525 } }
I20260924 21:11:56.503032 21849 raft_consensus.cc:388] T 00000000000000000000000000000000 P b9f5583bbcfb489e9ad752d58c1b1130 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 21:11:56.503053 21849 raft_consensus.cc:482] T 00000000000000000000000000000000 P b9f5583bbcfb489e9ad752d58c1b1130 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 21:11:56.503116 21849 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b9f5583bbcfb489e9ad752d58c1b1130 [term 0 FOLLOWER]: Advancing to term 1
I20260924 21:11:56.503803 21849 raft_consensus.cc:504] T 00000000000000000000000000000000 P b9f5583bbcfb489e9ad752d58c1b1130 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9f5583bbcfb489e9ad752d58c1b1130" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33525 } }
I20260924 21:11:56.503947 21849 leader_election.cc:302] T 00000000000000000000000000000000 P b9f5583bbcfb489e9ad752d58c1b1130 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b9f5583bbcfb489e9ad752d58c1b1130; no voters:
I20260924 21:11:56.504118 21849 leader_election.cc:288] T 00000000000000000000000000000000 P b9f5583bbcfb489e9ad752d58c1b1130 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 21:11:56.504141 21859 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b9f5583bbcfb489e9ad752d58c1b1130 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 21:11:56.504277 21859 raft_consensus.cc:686] T 00000000000000000000000000000000 P b9f5583bbcfb489e9ad752d58c1b1130 [term 1 LEADER]: Becoming Leader. State: Replica: b9f5583bbcfb489e9ad752d58c1b1130, State: Running, Role: LEADER
I20260924 21:11:56.504388 21849 sys_catalog.cc:564] T 00000000000000000000000000000000 P b9f5583bbcfb489e9ad752d58c1b1130 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 21:11:56.504900 21852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:11:56.503829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.21:42001 (local address 127.0.0.1:33525)
0924 21:11:56.503861 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0924 21:11:56.503863 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 21:11:56.503984 (+ 121us) server_negotiation.cc:407] Connection header received
0924 21:11:56.504077 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:11:56.504078 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:11:56.504082 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:11:56.504126 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:11:56.504651 (+ 525us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 21:11:56.504652 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 21:11:56.504652 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 21:11:56.504657 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 21:11:56.504667 (+ 10us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 21:11:56.504671 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 21:11:56.504751 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0924 21:11:56.504812 (+ 61us) server_negotiation.cc:299] Negotiation successful
0924 21:11:56.504825 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260924 21:11:56.504379 21859 consensus_queue.cc:237] T 00000000000000000000000000000000 P b9f5583bbcfb489e9ad752d58c1b1130 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9f5583bbcfb489e9ad752d58c1b1130" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33525 } }
I20260924 21:11:56.506222 21860 sys_catalog.cc:455] T 00000000000000000000000000000000 P b9f5583bbcfb489e9ad752d58c1b1130 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b9f5583bbcfb489e9ad752d58c1b1130. Latest consensus state: current_term: 1 leader_uuid: "b9f5583bbcfb489e9ad752d58c1b1130" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9f5583bbcfb489e9ad752d58c1b1130" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33525 } } }
I20260924 21:11:56.506392 21860 sys_catalog.cc:458] T 00000000000000000000000000000000 P b9f5583bbcfb489e9ad752d58c1b1130 [sys.catalog]: This master's current role is: LEADER
I20260924 21:11:56.506222 21861 sys_catalog.cc:455] T 00000000000000000000000000000000 P b9f5583bbcfb489e9ad752d58c1b1130 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b9f5583bbcfb489e9ad752d58c1b1130" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9f5583bbcfb489e9ad752d58c1b1130" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33525 } } }
W20260924 21:11:56.506678 21873 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b9f5583bbcfb489e9ad752d58c1b1130: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20260924 21:11:56.506709 21861 sys_catalog.cc:458] T 00000000000000000000000000000000 P b9f5583bbcfb489e9ad752d58c1b1130 [sys.catalog]: This master's current role is: LEADER
W20260924 21:11:56.506721 21873 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 21:11:56.506846 21874 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 21:11:56.507000 21874 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 21:11:56.508324 21874 catalog_manager.cc:1347] Generated new cluster ID: d2e55ff75e984e54b39302e102882f32
I20260924 21:11:56.508375 21874 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 21:11:56.519260 21874 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 21:11:56.519400 21874 catalog_manager.cc:1504] Loading token signing keys...
I20260924 21:11:56.528366 21874 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b9f5583bbcfb489e9ad752d58c1b1130: Generated new TSK 0
I20260924 21:11:56.528528 21874 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 21:11:56.531908 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAzNNjS/build/release/bin/kudu with pid 21781
2026-09-24T21:11:56Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1159 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790284076800039-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 21:11:56 dist-test-slave-c7tk krb5kdc[21880](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 21:11:56 dist-test-slave-c7tk krb5kdc[21880](info): set up 2 sockets
Sep 24 21:11:56 dist-test-slave-c7tk krb5kdc[21880](info): commencing operation
krb5kdc: starting...
W20260924 21:11:57.465373 392 mini_kdc.cc:121] Time spent starting KDC: real 0.916s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790284076800039-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790284076800039-392-0/krb5kdc/test-user.keytab.
Sep 24 21:11:57 dist-test-slave-c7tk krb5kdc[21880](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284317, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T21:11:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T21:11:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790284076800039-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790284076800039-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 21:11:57.566550 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAzNNjS/build/release/bin/kudu
/tmp/dist-test-taskAzNNjS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790284076800039-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790284076800039-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790284076800039-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790284076800039-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44525
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790284076800039-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38193
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44525
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790284076800039-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790284076800039-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790284076800039-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 21:11:57.644790 21893 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 21:11:57.645052 21893 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 21:11:57.645090 21893 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 21:11:57.646214 21893 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 21:11:57.646250 21893 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 21:11:57.646263 21893 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 21:11:57.646274 21893 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 21:11:57.646286 21893 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 21:11:57.647591 21893 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38193
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790284076800039-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790284076800039-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44525
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790284076800039-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44525
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790284076800039-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21893
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790284076800039-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 19:49:06 UTC on ccycloud.quasar-vhqjoc.root.comops.site
build id 4670864
I20260924 21:11:57.647809 21893 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 21:11:57.648054 21893 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 21:11:57.652686 21898 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 21:11:57.652722 21901 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 21:11:57.652762 21893 server_base.cc:1034] running on GCE node
W20260924 21:11:57.652705 21899 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 21:11:57.653112 21893 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 21:11:57.653407 21893 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 21:11:57.654556 21893 hybrid_clock.cc:648] HybridClock initialized: now 1790284317654535 us; error 32 us; skew 500 ppm
Sep 24 21:11:57 dist-test-slave-c7tk krb5kdc[21880](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284317, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 21:11:57.656615 21893 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 21:11:57.657052 21893 webserver.cc:460] Webserver started at http://127.0.98.62:33481/ using document root <none> and password file <none>
I20260924 21:11:57.657276 21893 fs_manager.cc:359] Metadata directory not provided
I20260924 21:11:57.657342 21893 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 21:11:57.657495 21893 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 21:11:57.658509 21893 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790284076800039-392-0/minicluster-data/master-0/data/instance:
uuid: "28878774a19548ec98010d69ad3b1341"
format_stamp: "Formatted at 2026-09-24 21:11:57 on dist-test-slave-c7tk"
server_key: "d37522ee8a7088388ba08d682f33fa07"
server_key_iv: "0a5a7df8fa27577536c15a7d32213ed4"
server_key_version: "encryptionkey@0"
I20260924 21:11:57.658805 21893 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790284076800039-392-0/minicluster-data/master-0/wal/instance:
uuid: "28878774a19548ec98010d69ad3b1341"
format_stamp: "Formatted at 2026-09-24 21:11:57 on dist-test-slave-c7tk"
server_key: "d37522ee8a7088388ba08d682f33fa07"
server_key_iv: "0a5a7df8fa27577536c15a7d32213ed4"
server_key_version: "encryptionkey@0"
I20260924 21:11:57.660028 21893 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260924 21:11:57.660797 21908 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 21:11:57.660975 21893 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 21:11:57.661072 21893 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790284076800039-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790284076800039-392-0/minicluster-data/master-0/wal
uuid: "28878774a19548ec98010d69ad3b1341"
format_stamp: "Formatted at 2026-09-24 21:11:57 on dist-test-slave-c7tk"
server_key: "d37522ee8a7088388ba08d682f33fa07"
server_key_iv: "0a5a7df8fa27577536c15a7d32213ed4"
server_key_version: "encryptionkey@0"
I20260924 21:11:57.661139 21893 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790284076800039-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790284076800039-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790284076800039-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 21:11:57.682238 21893 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 21:11:57.683832 21893 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 21:11:57.683991 21893 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 21:11:57.687659 21893 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44525
I20260924 21:11:57.687700 21960 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44525 every 8 connection(s)
I20260924 21:11:57.688009 21893 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790284076800039-392-0/minicluster-data/master-0/data/info.pb
I20260924 21:11:57.688766 21961 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 21:11:57.690409 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAzNNjS/build/release/bin/kudu as pid 21893
I20260924 21:11:57.690532 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790284076800039-392-0/minicluster-data/master-0/wal/instance
I20260924 21:11:57.690709 392 external_mini_cluster.cc:1428] Setting key f95f08c4a05aa212a18aa7420519d02d
I20260924 21:11:57.690891 21961 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 28878774a19548ec98010d69ad3b1341: Bootstrap starting.
I20260924 21:11:57.691576 21961 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 28878774a19548ec98010d69ad3b1341: Neither blocks nor log segments found. Creating new log.
I20260924 21:11:57.691869 21961 log.cc:824] T 00000000000000000000000000000000 P 28878774a19548ec98010d69ad3b1341: Log is configured to *not* fsync() on all Append() calls
I20260924 21:11:57.692628 21961 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 28878774a19548ec98010d69ad3b1341: No bootstrap required, opened a new log
I20260924 21:11:57.694551 21961 raft_consensus.cc:348] T 00000000000000000000000000000000 P 28878774a19548ec98010d69ad3b1341 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28878774a19548ec98010d69ad3b1341" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44525 } }
I20260924 21:11:57.694703 21961 raft_consensus.cc:374] T 00000000000000000000000000000000 P 28878774a19548ec98010d69ad3b1341 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 21:11:57.694744 21961 raft_consensus.cc:729] T 00000000000000000000000000000000 P 28878774a19548ec98010d69ad3b1341 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28878774a19548ec98010d69ad3b1341, State: Initialized, Role: FOLLOWER
I20260924 21:11:57.694875 21961 consensus_queue.cc:260] T 00000000000000000000000000000000 P 28878774a19548ec98010d69ad3b1341 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28878774a19548ec98010d69ad3b1341" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44525 } }
I20260924 21:11:57.694949 21961 raft_consensus.cc:388] T 00000000000000000000000000000000 P 28878774a19548ec98010d69ad3b1341 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 21:11:57.694968 21961 raft_consensus.cc:482] T 00000000000000000000000000000000 P 28878774a19548ec98010d69ad3b1341 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 21:11:57.695036 21961 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 28878774a19548ec98010d69ad3b1341 [term 0 FOLLOWER]: Advancing to term 1
I20260924 21:11:57.695603 21961 raft_consensus.cc:504] T 00000000000000000000000000000000 P 28878774a19548ec98010d69ad3b1341 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28878774a19548ec98010d69ad3b1341" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44525 } }
I20260924 21:11:57.695691 21961 leader_election.cc:302] T 00000000000000000000000000000000 P 28878774a19548ec98010d69ad3b1341 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 28878774a19548ec98010d69ad3b1341; no voters:
I20260924 21:11:57.695850 21961 leader_election.cc:288] T 00000000000000000000000000000000 P 28878774a19548ec98010d69ad3b1341 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 21:11:57.695890 21966 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 28878774a19548ec98010d69ad3b1341 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 21:11:57.696004 21966 raft_consensus.cc:686] T 00000000000000000000000000000000 P 28878774a19548ec98010d69ad3b1341 [term 1 LEADER]: Becoming Leader. State: Replica: 28878774a19548ec98010d69ad3b1341, State: Running, Role: LEADER
I20260924 21:11:57.696111 21961 sys_catalog.cc:564] T 00000000000000000000000000000000 P 28878774a19548ec98010d69ad3b1341 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 21:11:57.696147 21966 consensus_queue.cc:237] T 00000000000000000000000000000000 P 28878774a19548ec98010d69ad3b1341 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28878774a19548ec98010d69ad3b1341" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44525 } }
I20260924 21:11:57.696497 21966 sys_catalog.cc:455] T 00000000000000000000000000000000 P 28878774a19548ec98010d69ad3b1341 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 28878774a19548ec98010d69ad3b1341. Latest consensus state: current_term: 1 leader_uuid: "28878774a19548ec98010d69ad3b1341" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28878774a19548ec98010d69ad3b1341" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44525 } } }
I20260924 21:11:57.696590 21966 sys_catalog.cc:458] T 00000000000000000000000000000000 P 28878774a19548ec98010d69ad3b1341 [sys.catalog]: This master's current role is: LEADER
I20260924 21:11:57.696516 21967 sys_catalog.cc:455] T 00000000000000000000000000000000 P 28878774a19548ec98010d69ad3b1341 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "28878774a19548ec98010d69ad3b1341" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28878774a19548ec98010d69ad3b1341" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44525 } } }
I20260924 21:11:57.696735 21967 sys_catalog.cc:458] T 00000000000000000000000000000000 P 28878774a19548ec98010d69ad3b1341 [sys.catalog]: This master's current role is: LEADER
I20260924 21:11:57.696797 21974 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 21:11:57.697293 21974 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 24 21:11:57 dist-test-slave-c7tk krb5kdc[21880](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284317, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 21:11:57.698787 21974 catalog_manager.cc:1347] Generated new cluster ID: 96f257edf26c4833be3bee30bc0ce3ce
I20260924 21:11:57.698843 21974 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 21:11:57.701537 21964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:11:57.691170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38915 (local address 127.0.98.62:44525)
0924 21:11:57.691361 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0924 21:11:57.691369 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 21:11:57.691394 (+ 25us) server_negotiation.cc:407] Connection header received
0924 21:11:57.691829 (+ 435us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:11:57.691836 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:11:57.691886 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:11:57.692039 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:11:57.692810 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:11:57.693663 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:11:57.694755 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:11:57.695268 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:11:57.695339 (+ 71us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:11:57.698131 (+ 2792us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 21:11:57.698134 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 21:11:57.698134 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 21:11:57.698143 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 21:11:57.699876 (+ 1733us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:11:57.700318 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:11:57.700321 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:11:57.700321 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:11:57.700422 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:11:57.700671 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:11:57.700672 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:11:57.700673 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:11:57.700952 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 21:11:57.701103 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0924 21:11:57.701225 (+ 122us) server_negotiation.cc:299] Negotiation successful
0924 21:11:57.701320 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":151,"thread_start_us":113,"threads_started":1}
I20260924 21:11:57.702378 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 21:11:57.709213 21974 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 21:11:57.709875 21974 catalog_manager.cc:1504] Loading token signing keys...
I20260924 21:11:57.712357 21964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:11:57.703739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35367 (local address 127.0.98.62:44525)
0924 21:11:57.703769 (+ 30us) server_negotiation.cc:206] Beginning negotiation
0924 21:11:57.703770 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 21:11:57.705970 (+ 2200us) server_negotiation.cc:407] Connection header received
0924 21:11:57.706008 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:11:57.706009 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:11:57.706018 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:11:57.706066 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:11:57.706558 (+ 492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:11:57.707277 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:11:57.708193 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:11:57.708494 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:11:57.708558 (+ 64us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:11:57.709870 (+ 1312us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 21:11:57.709871 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 21:11:57.709871 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 21:11:57.709877 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 21:11:57.711366 (+ 1489us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:11:57.711705 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:11:57.711705 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:11:57.711706 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:11:57.711756 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:11:57.711939 (+ 183us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:11:57.711939 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:11:57.711940 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:11:57.712039 (+ 99us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 21:11:57.712122 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0924 21:11:57.712231 (+ 109us) server_negotiation.cc:299] Negotiation successful
0924 21:11:57.712284 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260924 21:11:57.713234 21974 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 28878774a19548ec98010d69ad3b1341: Generated new TSK 0
I20260924 21:11:57.713366 21974 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 21:11:57.718516 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAzNNjS/build/release/bin/kudu with pid 21893
W20260924 21:11:57.722394 21878 connection.cc:535] client connection to 127.0.98.62:44525 recv error: Network error: failed to read from TLS socket (remote: 127.0.98.62:44525): Connection reset by peer (error 104)
2026-09-24T21:11:57Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1186 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790284076800039-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 21:11:57 dist-test-slave-c7tk krb5kdc[21992](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 21:11:57 dist-test-slave-c7tk krb5kdc[21992](info): set up 2 sockets
Sep 24 21:11:57 dist-test-slave-c7tk krb5kdc[21992](info): commencing operation
krb5kdc: starting...
W20260924 21:11:58.677601 392 mini_kdc.cc:121] Time spent starting KDC: real 0.942s user 0.000s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790284076800039-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790284076800039-392-0/krb5kdc/test-user.keytab.
Sep 24 21:11:58 dist-test-slave-c7tk krb5kdc[21992](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284318, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T21:11:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T21:11:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790284076800039-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790284076800039-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 21:11:58.782198 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAzNNjS/build/release/bin/kudu
/tmp/dist-test-taskAzNNjS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790284076800039-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790284076800039-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790284076800039-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790284076800039-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35159
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790284076800039-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41359
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35159
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790284076800039-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790284076800039-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790284076800039-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 21:11:58.851168 22005 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 21:11:58.851370 22005 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 21:11:58.851416 22005 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 21:11:58.852455 22005 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 21:11:58.852512 22005 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 21:11:58.852535 22005 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 21:11:58.852560 22005 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 21:11:58.852582 22005 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 21:11:58.854024 22005 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41359
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790284076800039-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790284076800039-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35159
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790284076800039-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35159
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790284076800039-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22005
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790284076800039-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 19:49:06 UTC on ccycloud.quasar-vhqjoc.root.comops.site
build id 4670864
I20260924 21:11:58.854272 22005 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 21:11:58.854475 22005 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 21:11:58.858672 22011 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 21:11:58.858680 22010 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 21:11:58.858745 22013 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 21:11:58.858829 22005 server_base.cc:1034] running on GCE node
I20260924 21:11:58.859045 22005 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 21:11:58.859272 22005 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 21:11:58.860420 22005 hybrid_clock.cc:648] HybridClock initialized: now 1790284318860409 us; error 37 us; skew 500 ppm
Sep 24 21:11:58 dist-test-slave-c7tk krb5kdc[21992](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284318, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 21:11:58.862609 22005 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 21:11:58.862999 22005 webserver.cc:460] Webserver started at http://127.0.98.62:39191/ using document root <none> and password file <none>
I20260924 21:11:58.863179 22005 fs_manager.cc:359] Metadata directory not provided
I20260924 21:11:58.863234 22005 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 21:11:58.863359 22005 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 21:11:58.864173 22005 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790284076800039-392-0/minicluster-data/master-0/data/instance:
uuid: "35de6f7ce59642d69006245692e20c4c"
format_stamp: "Formatted at 2026-09-24 21:11:58 on dist-test-slave-c7tk"
server_key: "6d28bb571af18e8c28c3b75f1e2a3248"
server_key_iv: "9e676159a1536aa77cab1055e730501a"
server_key_version: "encryptionkey@0"
I20260924 21:11:58.864470 22005 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790284076800039-392-0/minicluster-data/master-0/wal/instance:
uuid: "35de6f7ce59642d69006245692e20c4c"
format_stamp: "Formatted at 2026-09-24 21:11:58 on dist-test-slave-c7tk"
server_key: "6d28bb571af18e8c28c3b75f1e2a3248"
server_key_iv: "9e676159a1536aa77cab1055e730501a"
server_key_version: "encryptionkey@0"
I20260924 21:11:58.865698 22005 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 21:11:58.866513 22020 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 21:11:58.866679 22005 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 21:11:58.866775 22005 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790284076800039-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790284076800039-392-0/minicluster-data/master-0/wal
uuid: "35de6f7ce59642d69006245692e20c4c"
format_stamp: "Formatted at 2026-09-24 21:11:58 on dist-test-slave-c7tk"
server_key: "6d28bb571af18e8c28c3b75f1e2a3248"
server_key_iv: "9e676159a1536aa77cab1055e730501a"
server_key_version: "encryptionkey@0"
I20260924 21:11:58.866863 22005 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790284076800039-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790284076800039-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790284076800039-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 21:11:58.891423 22005 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 21:11:58.893103 22005 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 21:11:58.893281 22005 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 21:11:58.897481 22005 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35159
I20260924 21:11:58.897507 22072 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35159 every 8 connection(s)
I20260924 21:11:58.897861 22005 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790284076800039-392-0/minicluster-data/master-0/data/info.pb
I20260924 21:11:58.898661 22073 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 21:11:58.900985 22073 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 35de6f7ce59642d69006245692e20c4c: Bootstrap starting.
I20260924 21:11:58.901539 22073 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 35de6f7ce59642d69006245692e20c4c: Neither blocks nor log segments found. Creating new log.
I20260924 21:11:58.901783 22073 log.cc:824] T 00000000000000000000000000000000 P 35de6f7ce59642d69006245692e20c4c: Log is configured to *not* fsync() on all Append() calls
I20260924 21:11:58.902420 22073 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 35de6f7ce59642d69006245692e20c4c: No bootstrap required, opened a new log
I20260924 21:11:58.903733 22073 raft_consensus.cc:348] T 00000000000000000000000000000000 P 35de6f7ce59642d69006245692e20c4c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35de6f7ce59642d69006245692e20c4c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35159 } }
I20260924 21:11:58.903841 22073 raft_consensus.cc:374] T 00000000000000000000000000000000 P 35de6f7ce59642d69006245692e20c4c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 21:11:58.903861 22073 raft_consensus.cc:729] T 00000000000000000000000000000000 P 35de6f7ce59642d69006245692e20c4c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 35de6f7ce59642d69006245692e20c4c, State: Initialized, Role: FOLLOWER
I20260924 21:11:58.903996 22073 consensus_queue.cc:260] T 00000000000000000000000000000000 P 35de6f7ce59642d69006245692e20c4c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35de6f7ce59642d69006245692e20c4c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35159 } }
I20260924 21:11:58.904069 22073 raft_consensus.cc:388] T 00000000000000000000000000000000 P 35de6f7ce59642d69006245692e20c4c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 21:11:58.904124 22073 raft_consensus.cc:482] T 00000000000000000000000000000000 P 35de6f7ce59642d69006245692e20c4c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 21:11:58.904181 22073 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 35de6f7ce59642d69006245692e20c4c [term 0 FOLLOWER]: Advancing to term 1
I20260924 21:11:58.904610 22073 raft_consensus.cc:504] T 00000000000000000000000000000000 P 35de6f7ce59642d69006245692e20c4c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35de6f7ce59642d69006245692e20c4c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35159 } }
I20260924 21:11:58.904681 22073 leader_election.cc:302] T 00000000000000000000000000000000 P 35de6f7ce59642d69006245692e20c4c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 35de6f7ce59642d69006245692e20c4c; no voters:
I20260924 21:11:58.904832 22073 leader_election.cc:288] T 00000000000000000000000000000000 P 35de6f7ce59642d69006245692e20c4c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 21:11:58.904860 22076 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 35de6f7ce59642d69006245692e20c4c [term 1 FOLLOWER]: Leader election won for term 1
I20260924 21:11:58.905061 22076 raft_consensus.cc:686] T 00000000000000000000000000000000 P 35de6f7ce59642d69006245692e20c4c [term 1 LEADER]: Becoming Leader. State: Replica: 35de6f7ce59642d69006245692e20c4c, State: Running, Role: LEADER
I20260924 21:11:58.905130 22073 sys_catalog.cc:564] T 00000000000000000000000000000000 P 35de6f7ce59642d69006245692e20c4c [sys.catalog]: configured and running, proceeding with master startup.
I20260924 21:11:58.905203 22076 consensus_queue.cc:237] T 00000000000000000000000000000000 P 35de6f7ce59642d69006245692e20c4c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35de6f7ce59642d69006245692e20c4c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35159 } }
I20260924 21:11:58.905633 22077 sys_catalog.cc:455] T 00000000000000000000000000000000 P 35de6f7ce59642d69006245692e20c4c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "35de6f7ce59642d69006245692e20c4c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35de6f7ce59642d69006245692e20c4c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35159 } } }
I20260924 21:11:58.905719 22077 sys_catalog.cc:458] T 00000000000000000000000000000000 P 35de6f7ce59642d69006245692e20c4c [sys.catalog]: This master's current role is: LEADER
I20260924 21:11:58.905844 22078 sys_catalog.cc:455] T 00000000000000000000000000000000 P 35de6f7ce59642d69006245692e20c4c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 35de6f7ce59642d69006245692e20c4c. Latest consensus state: current_term: 1 leader_uuid: "35de6f7ce59642d69006245692e20c4c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35de6f7ce59642d69006245692e20c4c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35159 } } }
I20260924 21:11:58.905920 22078 sys_catalog.cc:458] T 00000000000000000000000000000000 P 35de6f7ce59642d69006245692e20c4c [sys.catalog]: This master's current role is: LEADER
I20260924 21:11:58.905938 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAzNNjS/build/release/bin/kudu as pid 22005
I20260924 21:11:58.906038 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790284076800039-392-0/minicluster-data/master-0/wal/instance
I20260924 21:11:58.906215 392 external_mini_cluster.cc:1428] Setting key 4702917d30dba4a602e99d7534001862
I20260924 21:11:58.906243 22085 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 21:11:58.906715 22085 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 21:11:58.908015 22085 catalog_manager.cc:1347] Generated new cluster ID: ff0e64a9d371427a915a4202f83a5382
I20260924 21:11:58.908062 22085 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 21:11:58.911782 22095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:11:58.906798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57501 (local address 127.0.98.62:35159)
0924 21:11:58.906947 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0924 21:11:58.906955 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 21:11:58.906985 (+ 30us) server_negotiation.cc:407] Connection header received
0924 21:11:58.907204 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:11:58.907211 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:11:58.907262 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:11:58.907369 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:11:58.907620 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:11:58.908814 (+ 1194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:11:58.909791 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:11:58.910377 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:11:58.910477 (+ 100us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:11:58.911152 (+ 675us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 21:11:58.911156 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 21:11:58.911156 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 21:11:58.911365 (+ 209us) server_negotiation.cc:922] calling sasl_server_start()
0924 21:11:58.911397 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 21:11:58.911421 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 21:11:58.911506 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0924 21:11:58.911601 (+ 95us) server_negotiation.cc:299] Negotiation successful
0924 21:11:58.911620 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110,"thread_start_us":68,"threads_started":1}
I20260924 21:11:58.912710 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 21:11:58.921272 22095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:11:58.918113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56943 (local address 127.0.98.62:35159)
0924 21:11:58.918150 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0924 21:11:58.918151 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 21:11:58.918252 (+ 101us) server_negotiation.cc:407] Connection header received
0924 21:11:58.918283 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:11:58.918284 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:11:58.918296 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:11:58.918362 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:11:58.918511 (+ 149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:11:58.919203 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:11:58.920027 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:11:58.920345 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:11:58.920414 (+ 69us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:11:58.921040 (+ 626us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 21:11:58.921041 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 21:11:58.921042 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 21:11:58.921046 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 21:11:58.921060 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 21:11:58.921064 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 21:11:58.921129 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0924 21:11:58.921192 (+ 63us) server_negotiation.cc:299] Negotiation successful
0924 21:11:58.921203 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260924 21:11:58.927336 22085 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 21:11:58.927718 22085 catalog_manager.cc:1504] Loading token signing keys...
I20260924 21:11:58.931758 22085 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 35de6f7ce59642d69006245692e20c4c: Generated new TSK 0
I20260924 21:11:58.931900 22085 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 21:11:58.937381 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAzNNjS/build/release/bin/kudu with pid 22005
W20260924 21:11:58.941007 21990 connection.cc:535] client connection to 127.0.98.62:35159 recv error: Network error: failed to read from TLS socket (remote: 127.0.98.62:35159): Connection reset by peer (error 104)
2026-09-24T21:11:58Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1218 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790284076800039-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 21:11:58 dist-test-slave-c7tk krb5kdc[22104](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 21:11:58 dist-test-slave-c7tk krb5kdc[22104](info): set up 2 sockets
Sep 24 21:11:58 dist-test-slave-c7tk krb5kdc[22104](info): commencing operation
krb5kdc: starting...
W20260924 21:11:59.853617 392 mini_kdc.cc:121] Time spent starting KDC: real 0.899s user 0.000s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790284076800039-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790284076800039-392-0/krb5kdc/test-user.keytab.
Sep 24 21:11:59 dist-test-slave-c7tk krb5kdc[22104](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284319, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T21:11:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T21:11:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790284076800039-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790284076800039-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 21:11:59.952971 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAzNNjS/build/release/bin/kudu
/tmp/dist-test-taskAzNNjS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790284076800039-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790284076800039-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790284076800039-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790284076800039-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35549
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790284076800039-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46671
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35549
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790284076800039-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790284076800039-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790284076800039-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 21:12:00.023669 22117 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 21:12:00.023916 22117 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 21:12:00.023990 22117 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 21:12:00.025010 22117 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 21:12:00.025076 22117 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 21:12:00.025099 22117 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 21:12:00.025125 22117 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 21:12:00.025151 22117 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 21:12:00.026425 22117 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46671
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790284076800039-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790284076800039-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35549
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790284076800039-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35549
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790284076800039-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22117
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790284076800039-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 19:49:06 UTC on ccycloud.quasar-vhqjoc.root.comops.site
build id 4670864
I20260924 21:12:00.026655 22117 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 21:12:00.026842 22117 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 21:12:00.031129 22125 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 21:12:00.031136 22122 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 21:12:00.031235 22117 server_base.cc:1034] running on GCE node
W20260924 21:12:00.031235 22123 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 21:12:00.031541 22117 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 21:12:00.031773 22117 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 21:12:00.032912 22117 hybrid_clock.cc:648] HybridClock initialized: now 1790284320032888 us; error 42 us; skew 500 ppm
Sep 24 21:12:00 dist-test-slave-c7tk krb5kdc[22104](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284320, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 21:12:00.035187 22117 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 21:12:00.035564 22117 webserver.cc:460] Webserver started at http://127.0.98.62:39741/ using document root <none> and password file <none>
I20260924 21:12:00.035782 22117 fs_manager.cc:359] Metadata directory not provided
I20260924 21:12:00.035842 22117 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 21:12:00.035982 22117 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 21:12:00.036865 22117 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790284076800039-392-0/minicluster-data/master-0/data/instance:
uuid: "a1043a6ad8f14b0c8d1e8c6ed5686434"
format_stamp: "Formatted at 2026-09-24 21:12:00 on dist-test-slave-c7tk"
server_key: "6e1d2a7e2f3bb0e7962d73bd4be491b5"
server_key_iv: "edf55cdc6af4cb1716feffb41106e2af"
server_key_version: "encryptionkey@0"
I20260924 21:12:00.037189 22117 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790284076800039-392-0/minicluster-data/master-0/wal/instance:
uuid: "a1043a6ad8f14b0c8d1e8c6ed5686434"
format_stamp: "Formatted at 2026-09-24 21:12:00 on dist-test-slave-c7tk"
server_key: "6e1d2a7e2f3bb0e7962d73bd4be491b5"
server_key_iv: "edf55cdc6af4cb1716feffb41106e2af"
server_key_version: "encryptionkey@0"
I20260924 21:12:00.038400 22117 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 21:12:00.039040 22132 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 21:12:00.039181 22117 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 21:12:00.039261 22117 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790284076800039-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790284076800039-392-0/minicluster-data/master-0/wal
uuid: "a1043a6ad8f14b0c8d1e8c6ed5686434"
format_stamp: "Formatted at 2026-09-24 21:12:00 on dist-test-slave-c7tk"
server_key: "6e1d2a7e2f3bb0e7962d73bd4be491b5"
server_key_iv: "edf55cdc6af4cb1716feffb41106e2af"
server_key_version: "encryptionkey@0"
I20260924 21:12:00.039330 22117 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790284076800039-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790284076800039-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790284076800039-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 21:12:00.042280 22117 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 21:12:00.044543 22117 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 21:12:00.044673 22117 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 21:12:00.048722 22117 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35549
I20260924 21:12:00.048746 22184 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35549 every 8 connection(s)
I20260924 21:12:00.049009 22117 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790284076800039-392-0/minicluster-data/master-0/data/info.pb
I20260924 21:12:00.049839 22185 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 21:12:00.051707 22185 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a1043a6ad8f14b0c8d1e8c6ed5686434: Bootstrap starting.
I20260924 21:12:00.052130 22185 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a1043a6ad8f14b0c8d1e8c6ed5686434: Neither blocks nor log segments found. Creating new log.
I20260924 21:12:00.052320 22185 log.cc:824] T 00000000000000000000000000000000 P a1043a6ad8f14b0c8d1e8c6ed5686434: Log is configured to *not* fsync() on all Append() calls
I20260924 21:12:00.052842 22185 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a1043a6ad8f14b0c8d1e8c6ed5686434: No bootstrap required, opened a new log
I20260924 21:12:00.054255 22185 raft_consensus.cc:348] T 00000000000000000000000000000000 P a1043a6ad8f14b0c8d1e8c6ed5686434 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1043a6ad8f14b0c8d1e8c6ed5686434" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35549 } }
I20260924 21:12:00.054359 22185 raft_consensus.cc:374] T 00000000000000000000000000000000 P a1043a6ad8f14b0c8d1e8c6ed5686434 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 21:12:00.054378 22185 raft_consensus.cc:729] T 00000000000000000000000000000000 P a1043a6ad8f14b0c8d1e8c6ed5686434 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a1043a6ad8f14b0c8d1e8c6ed5686434, State: Initialized, Role: FOLLOWER
I20260924 21:12:00.054481 22185 consensus_queue.cc:260] T 00000000000000000000000000000000 P a1043a6ad8f14b0c8d1e8c6ed5686434 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1043a6ad8f14b0c8d1e8c6ed5686434" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35549 } }
I20260924 21:12:00.054549 22185 raft_consensus.cc:388] T 00000000000000000000000000000000 P a1043a6ad8f14b0c8d1e8c6ed5686434 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 21:12:00.054569 22185 raft_consensus.cc:482] T 00000000000000000000000000000000 P a1043a6ad8f14b0c8d1e8c6ed5686434 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 21:12:00.054590 22185 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a1043a6ad8f14b0c8d1e8c6ed5686434 [term 0 FOLLOWER]: Advancing to term 1
I20260924 21:12:00.054971 22185 raft_consensus.cc:504] T 00000000000000000000000000000000 P a1043a6ad8f14b0c8d1e8c6ed5686434 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1043a6ad8f14b0c8d1e8c6ed5686434" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35549 } }
I20260924 21:12:00.055047 22185 leader_election.cc:302] T 00000000000000000000000000000000 P a1043a6ad8f14b0c8d1e8c6ed5686434 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a1043a6ad8f14b0c8d1e8c6ed5686434; no voters:
I20260924 21:12:00.055145 22185 leader_election.cc:288] T 00000000000000000000000000000000 P a1043a6ad8f14b0c8d1e8c6ed5686434 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 21:12:00.055199 22188 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a1043a6ad8f14b0c8d1e8c6ed5686434 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 21:12:00.055353 22188 raft_consensus.cc:686] T 00000000000000000000000000000000 P a1043a6ad8f14b0c8d1e8c6ed5686434 [term 1 LEADER]: Becoming Leader. State: Replica: a1043a6ad8f14b0c8d1e8c6ed5686434, State: Running, Role: LEADER
I20260924 21:12:00.055352 22185 sys_catalog.cc:564] T 00000000000000000000000000000000 P a1043a6ad8f14b0c8d1e8c6ed5686434 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 21:12:00.055445 22188 consensus_queue.cc:237] T 00000000000000000000000000000000 P a1043a6ad8f14b0c8d1e8c6ed5686434 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1043a6ad8f14b0c8d1e8c6ed5686434" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35549 } }
I20260924 21:12:00.055774 22190 sys_catalog.cc:455] T 00000000000000000000000000000000 P a1043a6ad8f14b0c8d1e8c6ed5686434 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a1043a6ad8f14b0c8d1e8c6ed5686434. Latest consensus state: current_term: 1 leader_uuid: "a1043a6ad8f14b0c8d1e8c6ed5686434" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1043a6ad8f14b0c8d1e8c6ed5686434" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35549 } } }
I20260924 21:12:00.055856 22190 sys_catalog.cc:458] T 00000000000000000000000000000000 P a1043a6ad8f14b0c8d1e8c6ed5686434 [sys.catalog]: This master's current role is: LEADER
I20260924 21:12:00.055886 22189 sys_catalog.cc:455] T 00000000000000000000000000000000 P a1043a6ad8f14b0c8d1e8c6ed5686434 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a1043a6ad8f14b0c8d1e8c6ed5686434" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1043a6ad8f14b0c8d1e8c6ed5686434" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35549 } } }
I20260924 21:12:00.055948 22189 sys_catalog.cc:458] T 00000000000000000000000000000000 P a1043a6ad8f14b0c8d1e8c6ed5686434 [sys.catalog]: This master's current role is: LEADER
I20260924 21:12:00.056231 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAzNNjS/build/release/bin/kudu as pid 22117
I20260924 21:12:00.056344 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790284076800039-392-0/minicluster-data/master-0/wal/instance
I20260924 21:12:00.056501 392 external_mini_cluster.cc:1428] Setting key 4437005405119acdbc07599761cebb9f
I20260924 21:12:00.056972 22197 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 21:12:00.057492 22197 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 21:12:00.058641 22206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:12:00.057021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37033 (local address 127.0.98.62:35549)
0924 21:12:00.057174 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0924 21:12:00.057182 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 21:12:00.057207 (+ 25us) server_negotiation.cc:407] Connection header received
0924 21:12:00.057316 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:12:00.057323 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:12:00.057353 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:12:00.057475 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:12:00.058166 (+ 691us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 21:12:00.058168 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 21:12:00.058169 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 21:12:00.058172 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0924 21:12:00.058207 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 21:12:00.058219 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 21:12:00.058281 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0924 21:12:00.058426 (+ 145us) server_negotiation.cc:299] Negotiation successful
0924 21:12:00.058460 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110,"thread_start_us":60,"threads_started":1}
I20260924 21:12:00.058949 22197 catalog_manager.cc:1347] Generated new cluster ID: 5efa1d59bbb8499485ba4854638d1c99
I20260924 21:12:00.058992 22197 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 21:12:00.059396 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 21:12:00.070886 22206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:12:00.060659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45337 (local address 127.0.98.62:35549)
0924 21:12:00.060698 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0924 21:12:00.060700 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 21:12:00.069931 (+ 9231us) server_negotiation.cc:407] Connection header received
0924 21:12:00.069991 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:12:00.069991 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:12:00.069996 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:12:00.070060 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:12:00.070656 (+ 596us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 21:12:00.070658 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 21:12:00.070658 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 21:12:00.070662 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 21:12:00.070675 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 21:12:00.070679 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 21:12:00.070723 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0924 21:12:00.070782 (+ 59us) server_negotiation.cc:299] Negotiation successful
0924 21:12:00.070794 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260924 21:12:00.073150 22197 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 21:12:00.073277 22197 catalog_manager.cc:1504] Loading token signing keys...
I20260924 21:12:00.075635 22197 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a1043a6ad8f14b0c8d1e8c6ed5686434: Generated new TSK 0
I20260924 21:12:00.075766 22197 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 21:12:00.082084 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAzNNjS/build/release/bin/kudu with pid 22117
2026-09-24T21:12:00Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1144 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790284076800039-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 21:12:00 dist-test-slave-c7tk krb5kdc[22216](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 21:12:00 dist-test-slave-c7tk krb5kdc[22216](info): set up 2 sockets
Sep 24 21:12:00 dist-test-slave-c7tk krb5kdc[22216](info): commencing operation
krb5kdc: starting...
W20260924 21:12:01.013614 392 mini_kdc.cc:121] Time spent starting KDC: real 0.915s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790284076800039-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790284076800039-392-0/krb5kdc/test-user.keytab.
Sep 24 21:12:01 dist-test-slave-c7tk krb5kdc[22216](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284321, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T21:12:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T21:12:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790284076800039-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790284076800039-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 21:12:01.114481 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAzNNjS/build/release/bin/kudu
/tmp/dist-test-taskAzNNjS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790284076800039-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790284076800039-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790284076800039-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790284076800039-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44329
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790284076800039-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45447
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44329
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790284076800039-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790284076800039-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790284076800039-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 21:12:01.185288 22229 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 21:12:01.185449 22229 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 21:12:01.185468 22229 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 21:12:01.186712 22229 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 21:12:01.186765 22229 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 21:12:01.186779 22229 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 21:12:01.186791 22229 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 21:12:01.186853 22229 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 21:12:01.188050 22229 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45447
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790284076800039-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790284076800039-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44329
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790284076800039-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44329
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790284076800039-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22229
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790284076800039-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 19:49:06 UTC on ccycloud.quasar-vhqjoc.root.comops.site
build id 4670864
I20260924 21:12:01.188244 22229 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 21:12:01.188436 22229 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 21:12:01.192786 22237 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 21:12:01.192832 22235 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 21:12:01.192816 22229 server_base.cc:1034] running on GCE node
W20260924 21:12:01.192955 22234 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 21:12:01.193192 22229 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 21:12:01.193467 22229 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 21:12:01.194620 22229 hybrid_clock.cc:648] HybridClock initialized: now 1790284321194602 us; error 48 us; skew 500 ppm
Sep 24 21:12:01 dist-test-slave-c7tk krb5kdc[22216](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284321, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 21:12:01.196630 22229 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 21:12:01.197001 22229 webserver.cc:460] Webserver started at http://127.0.98.62:34203/ using document root <none> and password file <none>
I20260924 21:12:01.197206 22229 fs_manager.cc:359] Metadata directory not provided
I20260924 21:12:01.197268 22229 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 21:12:01.197402 22229 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 21:12:01.198338 22229 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790284076800039-392-0/minicluster-data/master-0/data/instance:
uuid: "de5c9d48f6114d20846d813370983d25"
format_stamp: "Formatted at 2026-09-24 21:12:01 on dist-test-slave-c7tk"
server_key: "78a57844086a806be50e7aa8cc7f9b46"
server_key_iv: "1ab2eb80919982df854e20d85b7bfa6a"
server_key_version: "encryptionkey@0"
I20260924 21:12:01.198673 22229 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790284076800039-392-0/minicluster-data/master-0/wal/instance:
uuid: "de5c9d48f6114d20846d813370983d25"
format_stamp: "Formatted at 2026-09-24 21:12:01 on dist-test-slave-c7tk"
server_key: "78a57844086a806be50e7aa8cc7f9b46"
server_key_iv: "1ab2eb80919982df854e20d85b7bfa6a"
server_key_version: "encryptionkey@0"
I20260924 21:12:01.199725 22229 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 21:12:01.200292 22244 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 21:12:01.200439 22229 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 21:12:01.200557 22229 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790284076800039-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790284076800039-392-0/minicluster-data/master-0/wal
uuid: "de5c9d48f6114d20846d813370983d25"
format_stamp: "Formatted at 2026-09-24 21:12:01 on dist-test-slave-c7tk"
server_key: "78a57844086a806be50e7aa8cc7f9b46"
server_key_iv: "1ab2eb80919982df854e20d85b7bfa6a"
server_key_version: "encryptionkey@0"
I20260924 21:12:01.200639 22229 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790284076800039-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790284076800039-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790284076800039-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 21:12:01.220464 22229 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 21:12:01.222076 22229 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 21:12:01.222230 22229 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 21:12:01.225344 22229 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44329
I20260924 21:12:01.225382 22296 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44329 every 8 connection(s)
I20260924 21:12:01.225641 22229 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790284076800039-392-0/minicluster-data/master-0/data/info.pb
I20260924 21:12:01.226313 22297 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 21:12:01.227963 22297 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P de5c9d48f6114d20846d813370983d25: Bootstrap starting.
I20260924 21:12:01.228106 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAzNNjS/build/release/bin/kudu as pid 22229
I20260924 21:12:01.228205 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790284076800039-392-0/minicluster-data/master-0/wal/instance
I20260924 21:12:01.228344 392 external_mini_cluster.cc:1428] Setting key 528f526e2240aa41cf245082e655b16c
I20260924 21:12:01.228514 22297 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P de5c9d48f6114d20846d813370983d25: Neither blocks nor log segments found. Creating new log.
I20260924 21:12:01.228803 22297 log.cc:824] T 00000000000000000000000000000000 P de5c9d48f6114d20846d813370983d25: Log is configured to *not* fsync() on all Append() calls
I20260924 21:12:01.229406 22297 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P de5c9d48f6114d20846d813370983d25: No bootstrap required, opened a new log
I20260924 21:12:01.231076 22297 raft_consensus.cc:348] T 00000000000000000000000000000000 P de5c9d48f6114d20846d813370983d25 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de5c9d48f6114d20846d813370983d25" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44329 } }
I20260924 21:12:01.231215 22297 raft_consensus.cc:374] T 00000000000000000000000000000000 P de5c9d48f6114d20846d813370983d25 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 21:12:01.231261 22297 raft_consensus.cc:729] T 00000000000000000000000000000000 P de5c9d48f6114d20846d813370983d25 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: de5c9d48f6114d20846d813370983d25, State: Initialized, Role: FOLLOWER
I20260924 21:12:01.231381 22297 consensus_queue.cc:260] T 00000000000000000000000000000000 P de5c9d48f6114d20846d813370983d25 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de5c9d48f6114d20846d813370983d25" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44329 } }
I20260924 21:12:01.231472 22297 raft_consensus.cc:388] T 00000000000000000000000000000000 P de5c9d48f6114d20846d813370983d25 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 21:12:01.231520 22297 raft_consensus.cc:482] T 00000000000000000000000000000000 P de5c9d48f6114d20846d813370983d25 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 21:12:01.231590 22297 raft_consensus.cc:3037] T 00000000000000000000000000000000 P de5c9d48f6114d20846d813370983d25 [term 0 FOLLOWER]: Advancing to term 1
I20260924 21:12:01.232142 22297 raft_consensus.cc:504] T 00000000000000000000000000000000 P de5c9d48f6114d20846d813370983d25 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de5c9d48f6114d20846d813370983d25" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44329 } }
I20260924 21:12:01.232251 22297 leader_election.cc:302] T 00000000000000000000000000000000 P de5c9d48f6114d20846d813370983d25 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: de5c9d48f6114d20846d813370983d25; no voters:
I20260924 21:12:01.232424 22297 leader_election.cc:288] T 00000000000000000000000000000000 P de5c9d48f6114d20846d813370983d25 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 21:12:01.232460 22302 raft_consensus.cc:2781] T 00000000000000000000000000000000 P de5c9d48f6114d20846d813370983d25 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 21:12:01.232626 22302 raft_consensus.cc:686] T 00000000000000000000000000000000 P de5c9d48f6114d20846d813370983d25 [term 1 LEADER]: Becoming Leader. State: Replica: de5c9d48f6114d20846d813370983d25, State: Running, Role: LEADER
I20260924 21:12:01.232684 22297 sys_catalog.cc:564] T 00000000000000000000000000000000 P de5c9d48f6114d20846d813370983d25 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 21:12:01.232786 22302 consensus_queue.cc:237] T 00000000000000000000000000000000 P de5c9d48f6114d20846d813370983d25 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de5c9d48f6114d20846d813370983d25" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44329 } }
I20260924 21:12:01.233142 22303 sys_catalog.cc:455] T 00000000000000000000000000000000 P de5c9d48f6114d20846d813370983d25 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "de5c9d48f6114d20846d813370983d25" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de5c9d48f6114d20846d813370983d25" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44329 } } }
I20260924 21:12:01.233225 22303 sys_catalog.cc:458] T 00000000000000000000000000000000 P de5c9d48f6114d20846d813370983d25 [sys.catalog]: This master's current role is: LEADER
I20260924 21:12:01.233349 22304 sys_catalog.cc:455] T 00000000000000000000000000000000 P de5c9d48f6114d20846d813370983d25 [sys.catalog]: SysCatalogTable state changed. Reason: New leader de5c9d48f6114d20846d813370983d25. Latest consensus state: current_term: 1 leader_uuid: "de5c9d48f6114d20846d813370983d25" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de5c9d48f6114d20846d813370983d25" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44329 } } }
I20260924 21:12:01.233394 22306 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 21:12:01.233409 22304 sys_catalog.cc:458] T 00000000000000000000000000000000 P de5c9d48f6114d20846d813370983d25 [sys.catalog]: This master's current role is: LEADER
I20260924 21:12:01.233901 22306 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 24 21:12:01 dist-test-slave-c7tk krb5kdc[22216](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284321, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 21:12:01.235246 22306 catalog_manager.cc:1347] Generated new cluster ID: 9a1efdb371a84285ae0dbfccbf906d50
I20260924 21:12:01.235304 22306 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 21:12:01.238164 22300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:12:01.228769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57721 (local address 127.0.98.62:44329)
0924 21:12:01.228973 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0924 21:12:01.228981 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 21:12:01.229004 (+ 23us) server_negotiation.cc:407] Connection header received
0924 21:12:01.229341 (+ 337us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:12:01.229346 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:12:01.229382 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:12:01.229517 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:12:01.230124 (+ 607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:12:01.230965 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:12:01.231948 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:12:01.232454 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:12:01.232900 (+ 446us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:12:01.235209 (+ 2309us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 21:12:01.235212 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 21:12:01.235213 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 21:12:01.235222 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 21:12:01.236701 (+ 1479us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:12:01.237032 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:12:01.237034 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:12:01.237034 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:12:01.237111 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:12:01.237293 (+ 182us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:12:01.237294 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:12:01.237294 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:12:01.237610 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 21:12:01.237770 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0924 21:12:01.237862 (+ 92us) server_negotiation.cc:299] Negotiation successful
0924 21:12:01.237971 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":164,"thread_start_us":75,"threads_started":1}
I20260924 21:12:01.238921 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 21:12:01.245296 22300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:12:01.240078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50571 (local address 127.0.98.62:44329)
0924 21:12:01.240104 (+ 26us) server_negotiation.cc:206] Beginning negotiation
0924 21:12:01.240106 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 21:12:01.240237 (+ 131us) server_negotiation.cc:407] Connection header received
0924 21:12:01.240294 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:12:01.240295 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:12:01.240303 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:12:01.240348 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:12:01.240869 (+ 521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:12:01.241321 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:12:01.242044 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:12:01.242215 (+ 171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:12:01.242286 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:12:01.243223 (+ 937us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 21:12:01.243225 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 21:12:01.243225 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 21:12:01.243230 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 21:12:01.244357 (+ 1127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:12:01.244663 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:12:01.244664 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:12:01.244664 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:12:01.244707 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:12:01.244880 (+ 173us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:12:01.244881 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:12:01.244881 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:12:01.244974 (+ 93us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 21:12:01.245070 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0924 21:12:01.245161 (+ 91us) server_negotiation.cc:299] Negotiation successful
0924 21:12:01.245220 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12}
I20260924 21:12:01.251636 22306 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 21:12:01.252068 22306 catalog_manager.cc:1504] Loading token signing keys...
I20260924 21:12:01.257259 22306 catalog_manager.cc:5868] T 00000000000000000000000000000000 P de5c9d48f6114d20846d813370983d25: Generated new TSK 0
I20260924 21:12:01.257400 22306 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 21:12:01.259385 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAzNNjS/build/release/bin/kudu with pid 22229
2026-09-24T21:12:01Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1177 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790284076800039-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 21:12:01 dist-test-slave-c7tk krb5kdc[22328](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 21:12:01 dist-test-slave-c7tk krb5kdc[22328](info): set up 2 sockets
Sep 24 21:12:01 dist-test-slave-c7tk krb5kdc[22328](info): commencing operation
krb5kdc: starting...
W20260924 21:12:02.194159 392 mini_kdc.cc:121] Time spent starting KDC: real 0.918s user 0.001s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790284076800039-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790284076800039-392-0/krb5kdc/test-user.keytab.
Sep 24 21:12:02 dist-test-slave-c7tk krb5kdc[22328](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284322, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T21:12:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T21:12:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790284076800039-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790284076800039-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 21:12:02.294759 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAzNNjS/build/release/bin/kudu
/tmp/dist-test-taskAzNNjS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790284076800039-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790284076800039-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790284076800039-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790284076800039-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39185
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790284076800039-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38323
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39185
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790284076800039-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790284076800039-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790284076800039-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 21:12:02.366093 22341 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 21:12:02.366251 22341 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 21:12:02.366271 22341 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 21:12:02.367326 22341 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 21:12:02.367360 22341 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 21:12:02.367374 22341 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 21:12:02.367386 22341 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 21:12:02.367398 22341 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 21:12:02.368690 22341 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38323
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790284076800039-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790284076800039-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39185
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790284076800039-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39185
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790284076800039-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22341
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790284076800039-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 19:49:06 UTC on ccycloud.quasar-vhqjoc.root.comops.site
build id 4670864
I20260924 21:12:02.368842 22341 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 21:12:02.368988 22341 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 21:12:02.373266 22349 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 21:12:02.373423 22341 server_base.cc:1034] running on GCE node
W20260924 21:12:02.373292 22346 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 21:12:02.373274 22347 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 21:12:02.373735 22341 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 21:12:02.374013 22341 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 21:12:02.375166 22341 hybrid_clock.cc:648] HybridClock initialized: now 1790284322375163 us; error 31 us; skew 500 ppm
Sep 24 21:12:02 dist-test-slave-c7tk krb5kdc[22328](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284322, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 21:12:02.384218 22341 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 21:12:02.384608 22341 webserver.cc:460] Webserver started at http://127.0.98.62:38835/ using document root <none> and password file <none>
I20260924 21:12:02.384789 22341 fs_manager.cc:359] Metadata directory not provided
I20260924 21:12:02.384845 22341 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 21:12:02.384966 22341 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 21:12:02.385901 22341 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790284076800039-392-0/minicluster-data/master-0/data/instance:
uuid: "1ce76a31a84e406caa6b747346d0ad4b"
format_stamp: "Formatted at 2026-09-24 21:12:02 on dist-test-slave-c7tk"
server_key: "e490570bf4a9e31071e29a5206345e82"
server_key_iv: "96df43673f3f3fdcd52f070ded37493f"
server_key_version: "encryptionkey@0"
I20260924 21:12:02.386209 22341 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790284076800039-392-0/minicluster-data/master-0/wal/instance:
uuid: "1ce76a31a84e406caa6b747346d0ad4b"
format_stamp: "Formatted at 2026-09-24 21:12:02 on dist-test-slave-c7tk"
server_key: "e490570bf4a9e31071e29a5206345e82"
server_key_iv: "96df43673f3f3fdcd52f070ded37493f"
server_key_version: "encryptionkey@0"
I20260924 21:12:02.387399 22341 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 21:12:02.387993 22356 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 21:12:02.388271 22341 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260924 21:12:02.388370 22341 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790284076800039-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790284076800039-392-0/minicluster-data/master-0/wal
uuid: "1ce76a31a84e406caa6b747346d0ad4b"
format_stamp: "Formatted at 2026-09-24 21:12:02 on dist-test-slave-c7tk"
server_key: "e490570bf4a9e31071e29a5206345e82"
server_key_iv: "96df43673f3f3fdcd52f070ded37493f"
server_key_version: "encryptionkey@0"
I20260924 21:12:02.388444 22341 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790284076800039-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790284076800039-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790284076800039-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 21:12:02.398185 22341 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 21:12:02.400125 22341 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 21:12:02.400269 22341 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 21:12:02.403867 22341 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39185
I20260924 21:12:02.403900 22408 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39185 every 8 connection(s)
I20260924 21:12:02.404186 22341 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790284076800039-392-0/minicluster-data/master-0/data/info.pb
I20260924 21:12:02.404852 22409 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 21:12:02.406733 22409 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1ce76a31a84e406caa6b747346d0ad4b: Bootstrap starting.
I20260924 21:12:02.407248 22409 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1ce76a31a84e406caa6b747346d0ad4b: Neither blocks nor log segments found. Creating new log.
I20260924 21:12:02.407493 22409 log.cc:824] T 00000000000000000000000000000000 P 1ce76a31a84e406caa6b747346d0ad4b: Log is configured to *not* fsync() on all Append() calls
I20260924 21:12:02.408107 22409 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1ce76a31a84e406caa6b747346d0ad4b: No bootstrap required, opened a new log
I20260924 21:12:02.408609 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAzNNjS/build/release/bin/kudu as pid 22341
I20260924 21:12:02.408715 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790284076800039-392-0/minicluster-data/master-0/wal/instance
I20260924 21:12:02.408869 392 external_mini_cluster.cc:1428] Setting key ceba7d21de83c93a5bc8b0782c1e74a8
I20260924 21:12:02.410039 22409 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1ce76a31a84e406caa6b747346d0ad4b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ce76a31a84e406caa6b747346d0ad4b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39185 } }
I20260924 21:12:02.410175 22409 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1ce76a31a84e406caa6b747346d0ad4b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 21:12:02.410208 22409 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1ce76a31a84e406caa6b747346d0ad4b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ce76a31a84e406caa6b747346d0ad4b, State: Initialized, Role: FOLLOWER
I20260924 21:12:02.410317 22409 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1ce76a31a84e406caa6b747346d0ad4b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ce76a31a84e406caa6b747346d0ad4b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39185 } }
I20260924 21:12:02.410399 22409 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1ce76a31a84e406caa6b747346d0ad4b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 21:12:02.410422 22409 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1ce76a31a84e406caa6b747346d0ad4b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 21:12:02.410485 22409 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1ce76a31a84e406caa6b747346d0ad4b [term 0 FOLLOWER]: Advancing to term 1
I20260924 21:12:02.411003 22409 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1ce76a31a84e406caa6b747346d0ad4b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ce76a31a84e406caa6b747346d0ad4b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39185 } }
I20260924 21:12:02.411080 22409 leader_election.cc:302] T 00000000000000000000000000000000 P 1ce76a31a84e406caa6b747346d0ad4b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1ce76a31a84e406caa6b747346d0ad4b; no voters:
I20260924 21:12:02.411259 22409 leader_election.cc:288] T 00000000000000000000000000000000 P 1ce76a31a84e406caa6b747346d0ad4b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 21:12:02.411283 22414 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1ce76a31a84e406caa6b747346d0ad4b [term 1 FOLLOWER]: Leader election won for term 1
I20260924 21:12:02.411387 22414 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1ce76a31a84e406caa6b747346d0ad4b [term 1 LEADER]: Becoming Leader. State: Replica: 1ce76a31a84e406caa6b747346d0ad4b, State: Running, Role: LEADER
I20260924 21:12:02.411506 22409 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1ce76a31a84e406caa6b747346d0ad4b [sys.catalog]: configured and running, proceeding with master startup.
I20260924 21:12:02.411525 22414 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1ce76a31a84e406caa6b747346d0ad4b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ce76a31a84e406caa6b747346d0ad4b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39185 } }
I20260924 21:12:02.411890 22415 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1ce76a31a84e406caa6b747346d0ad4b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1ce76a31a84e406caa6b747346d0ad4b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ce76a31a84e406caa6b747346d0ad4b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39185 } } }
I20260924 21:12:02.412104 22415 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1ce76a31a84e406caa6b747346d0ad4b [sys.catalog]: This master's current role is: LEADER
I20260924 21:12:02.412039 22416 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1ce76a31a84e406caa6b747346d0ad4b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1ce76a31a84e406caa6b747346d0ad4b. Latest consensus state: current_term: 1 leader_uuid: "1ce76a31a84e406caa6b747346d0ad4b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ce76a31a84e406caa6b747346d0ad4b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39185 } } }
I20260924 21:12:02.412322 22416 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1ce76a31a84e406caa6b747346d0ad4b [sys.catalog]: This master's current role is: LEADER
I20260924 21:12:02.412465 22423 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 21:12:02.412989 22423 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 21:12:02.414538 22423 catalog_manager.cc:1347] Generated new cluster ID: ed19d968734e4891bbd949d88afd0973
I20260924 21:12:02.414592 22423 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 21:12:02.414817 22413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:12:02.409301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36685 (local address 127.0.98.62:39185)
0924 21:12:02.409448 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0924 21:12:02.409455 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 21:12:02.409482 (+ 27us) server_negotiation.cc:407] Connection header received
0924 21:12:02.410015 (+ 533us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:12:02.410020 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:12:02.410058 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:12:02.410151 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:12:02.410363 (+ 212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:12:02.411698 (+ 1335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:12:02.412795 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:12:02.413259 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:12:02.413372 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:12:02.414066 (+ 694us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 21:12:02.414070 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 21:12:02.414070 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 21:12:02.414259 (+ 189us) server_negotiation.cc:922] calling sasl_server_start()
0924 21:12:02.414298 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 21:12:02.414311 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 21:12:02.414368 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0924 21:12:02.414564 (+ 196us) server_negotiation.cc:299] Negotiation successful
0924 21:12:02.414623 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94,"thread_start_us":57,"threads_started":1}
I20260924 21:12:02.418885 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 21:12:02.428776 22423 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 21:12:02.429174 22423 catalog_manager.cc:1504] Loading token signing keys...
I20260924 21:12:02.431452 22423 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1ce76a31a84e406caa6b747346d0ad4b: Generated new TSK 0
I20260924 21:12:02.431602 22423 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 21:12:02.431906 22413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:12:02.420125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40513 (local address 127.0.98.62:39185)
0924 21:12:02.420157 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0924 21:12:02.420158 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 21:12:02.428572 (+ 8414us) server_negotiation.cc:407] Connection header received
0924 21:12:02.428634 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:12:02.428634 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:12:02.428652 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:12:02.428692 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:12:02.428841 (+ 149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:12:02.429438 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:12:02.430323 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:12:02.430594 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:12:02.430662 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:12:02.431344 (+ 682us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 21:12:02.431345 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 21:12:02.431345 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 21:12:02.431350 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 21:12:02.431363 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 21:12:02.431368 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 21:12:02.431435 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0924 21:12:02.431794 (+ 359us) server_negotiation.cc:299] Negotiation successful
0924 21:12:02.431822 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260924 21:12:02.433568 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAzNNjS/build/release/bin/kudu with pid 22341
2026-09-24T21:12:02Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1174 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790284076800039-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 21:12:02 dist-test-slave-c7tk krb5kdc[22440](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 21:12:02 dist-test-slave-c7tk krb5kdc[22440](info): set up 2 sockets
Sep 24 21:12:02 dist-test-slave-c7tk krb5kdc[22440](info): commencing operation
krb5kdc: starting...
W20260924 21:12:03.377553 392 mini_kdc.cc:121] Time spent starting KDC: real 0.926s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790284076800039-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790284076800039-392-0/krb5kdc/test-user.keytab.
Sep 24 21:12:03 dist-test-slave-c7tk krb5kdc[22440](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284323, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T21:12:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T21:12:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790284076800039-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790284076800039-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 21:12:03.479758 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAzNNjS/build/release/bin/kudu
/tmp/dist-test-taskAzNNjS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790284076800039-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790284076800039-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790284076800039-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790284076800039-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35675
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790284076800039-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45567
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35675
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790284076800039-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790284076800039-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790284076800039-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 21:12:03.552218 22453 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 21:12:03.552380 22453 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 21:12:03.552416 22453 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 21:12:03.553445 22453 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 21:12:03.553481 22453 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 21:12:03.553494 22453 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 21:12:03.553507 22453 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 21:12:03.553519 22453 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 21:12:03.554862 22453 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45567
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790284076800039-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790284076800039-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35675
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790284076800039-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35675
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790284076800039-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22453
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790284076800039-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 19:49:06 UTC on ccycloud.quasar-vhqjoc.root.comops.site
build id 4670864
I20260924 21:12:03.555042 22453 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 21:12:03.555190 22453 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 21:12:03.559298 22461 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 21:12:03.559419 22453 server_base.cc:1034] running on GCE node
W20260924 21:12:03.559289 22459 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 21:12:03.559290 22458 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 21:12:03.559732 22453 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 21:12:03.559973 22453 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 21:12:03.561111 22453 hybrid_clock.cc:648] HybridClock initialized: now 1790284323561098 us; error 27 us; skew 500 ppm
Sep 24 21:12:03 dist-test-slave-c7tk krb5kdc[22440](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284323, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 21:12:03.563405 22453 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 21:12:03.563851 22453 webserver.cc:460] Webserver started at http://127.0.98.62:46549/ using document root <none> and password file <none>
I20260924 21:12:03.564059 22453 fs_manager.cc:359] Metadata directory not provided
I20260924 21:12:03.564129 22453 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 21:12:03.564272 22453 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 21:12:03.565222 22453 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790284076800039-392-0/minicluster-data/master-0/data/instance:
uuid: "3277dfd00f8a463792b3f2d73ff03622"
format_stamp: "Formatted at 2026-09-24 21:12:03 on dist-test-slave-c7tk"
server_key: "59c232b3ca439080dc69e4451b1d88a8"
server_key_iv: "2295080e39e4ecf23ef7372147ec487b"
server_key_version: "encryptionkey@0"
I20260924 21:12:03.565557 22453 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790284076800039-392-0/minicluster-data/master-0/wal/instance:
uuid: "3277dfd00f8a463792b3f2d73ff03622"
format_stamp: "Formatted at 2026-09-24 21:12:03 on dist-test-slave-c7tk"
server_key: "59c232b3ca439080dc69e4451b1d88a8"
server_key_iv: "2295080e39e4ecf23ef7372147ec487b"
server_key_version: "encryptionkey@0"
I20260924 21:12:03.566928 22453 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 21:12:03.567584 22468 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 21:12:03.567751 22453 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 21:12:03.567843 22453 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790284076800039-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790284076800039-392-0/minicluster-data/master-0/wal
uuid: "3277dfd00f8a463792b3f2d73ff03622"
format_stamp: "Formatted at 2026-09-24 21:12:03 on dist-test-slave-c7tk"
server_key: "59c232b3ca439080dc69e4451b1d88a8"
server_key_iv: "2295080e39e4ecf23ef7372147ec487b"
server_key_version: "encryptionkey@0"
I20260924 21:12:03.567924 22453 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790284076800039-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790284076800039-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790284076800039-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 21:12:03.571058 22453 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 21:12:03.572700 22453 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 21:12:03.572839 22453 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 21:12:03.576680 22453 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35675
I20260924 21:12:03.576709 22520 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35675 every 8 connection(s)
I20260924 21:12:03.577005 22453 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790284076800039-392-0/minicluster-data/master-0/data/info.pb
I20260924 21:12:03.577961 22521 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 21:12:03.579847 22521 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3277dfd00f8a463792b3f2d73ff03622: Bootstrap starting.
I20260924 21:12:03.580299 22521 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3277dfd00f8a463792b3f2d73ff03622: Neither blocks nor log segments found. Creating new log.
I20260924 21:12:03.580508 22521 log.cc:824] T 00000000000000000000000000000000 P 3277dfd00f8a463792b3f2d73ff03622: Log is configured to *not* fsync() on all Append() calls
I20260924 21:12:03.581105 22521 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3277dfd00f8a463792b3f2d73ff03622: No bootstrap required, opened a new log
I20260924 21:12:03.582573 22521 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3277dfd00f8a463792b3f2d73ff03622 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3277dfd00f8a463792b3f2d73ff03622" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35675 } }
I20260924 21:12:03.582684 22521 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3277dfd00f8a463792b3f2d73ff03622 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 21:12:03.582705 22521 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3277dfd00f8a463792b3f2d73ff03622 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3277dfd00f8a463792b3f2d73ff03622, State: Initialized, Role: FOLLOWER
I20260924 21:12:03.582814 22521 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3277dfd00f8a463792b3f2d73ff03622 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3277dfd00f8a463792b3f2d73ff03622" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35675 } }
I20260924 21:12:03.582878 22521 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3277dfd00f8a463792b3f2d73ff03622 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 21:12:03.582898 22521 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3277dfd00f8a463792b3f2d73ff03622 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 21:12:03.582922 22521 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3277dfd00f8a463792b3f2d73ff03622 [term 0 FOLLOWER]: Advancing to term 1
I20260924 21:12:03.583350 22521 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3277dfd00f8a463792b3f2d73ff03622 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3277dfd00f8a463792b3f2d73ff03622" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35675 } }
I20260924 21:12:03.583451 22521 leader_election.cc:302] T 00000000000000000000000000000000 P 3277dfd00f8a463792b3f2d73ff03622 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3277dfd00f8a463792b3f2d73ff03622; no voters:
I20260924 21:12:03.583581 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAzNNjS/build/release/bin/kudu as pid 22453
I20260924 21:12:03.583609 22521 leader_election.cc:288] T 00000000000000000000000000000000 P 3277dfd00f8a463792b3f2d73ff03622 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 21:12:03.583652 22524 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3277dfd00f8a463792b3f2d73ff03622 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 21:12:03.583702 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790284076800039-392-0/minicluster-data/master-0/wal/instance
I20260924 21:12:03.583799 22524 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3277dfd00f8a463792b3f2d73ff03622 [term 1 LEADER]: Becoming Leader. State: Replica: 3277dfd00f8a463792b3f2d73ff03622, State: Running, Role: LEADER
I20260924 21:12:03.583822 22521 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3277dfd00f8a463792b3f2d73ff03622 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 21:12:03.583859 392 external_mini_cluster.cc:1428] Setting key 73e81899e069baaaf643ce6f3137a282
I20260924 21:12:03.583918 22524 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3277dfd00f8a463792b3f2d73ff03622 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3277dfd00f8a463792b3f2d73ff03622" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35675 } }
I20260924 21:12:03.584288 22525 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3277dfd00f8a463792b3f2d73ff03622 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3277dfd00f8a463792b3f2d73ff03622" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3277dfd00f8a463792b3f2d73ff03622" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35675 } } }
I20260924 21:12:03.584474 22525 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3277dfd00f8a463792b3f2d73ff03622 [sys.catalog]: This master's current role is: LEADER
I20260924 21:12:03.584416 22526 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3277dfd00f8a463792b3f2d73ff03622 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3277dfd00f8a463792b3f2d73ff03622. Latest consensus state: current_term: 1 leader_uuid: "3277dfd00f8a463792b3f2d73ff03622" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3277dfd00f8a463792b3f2d73ff03622" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35675 } } }
I20260924 21:12:03.584666 22526 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3277dfd00f8a463792b3f2d73ff03622 [sys.catalog]: This master's current role is: LEADER
I20260924 21:12:03.585093 22530 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 21:12:03.585556 22530 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 21:12:03.586208 22529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:12:03.584302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37599 (local address 127.0.98.62:35675)
0924 21:12:03.584470 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0924 21:12:03.584477 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 21:12:03.584505 (+ 28us) server_negotiation.cc:407] Connection header received
0924 21:12:03.584626 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:12:03.584632 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:12:03.584691 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:12:03.584865 (+ 174us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:12:03.585762 (+ 897us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 21:12:03.585765 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 21:12:03.585765 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 21:12:03.585769 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 21:12:03.585842 (+ 73us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 21:12:03.585855 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 21:12:03.585913 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0924 21:12:03.586018 (+ 105us) server_negotiation.cc:299] Negotiation successful
0924 21:12:03.586051 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":55,"threads_started":1}
I20260924 21:12:03.587002 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 21:12:03.587067 22530 catalog_manager.cc:1347] Generated new cluster ID: 7e4f735d29634c318cb8aa9d833593f1
I20260924 21:12:03.587097 22530 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 21:12:03.590828 22529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:12:03.588429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56967 (local address 127.0.98.62:35675)
0924 21:12:03.588459 (+ 30us) server_negotiation.cc:206] Beginning negotiation
0924 21:12:03.588461 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 21:12:03.589982 (+ 1521us) server_negotiation.cc:407] Connection header received
0924 21:12:03.590030 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:12:03.590031 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:12:03.590036 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:12:03.590106 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:12:03.590617 (+ 511us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 21:12:03.590618 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 21:12:03.590619 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 21:12:03.590622 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0924 21:12:03.590636 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 21:12:03.590640 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 21:12:03.590702 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0924 21:12:03.590757 (+ 55us) server_negotiation.cc:299] Negotiation successful
0924 21:12:03.590768 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260924 21:12:03.601683 22530 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 21:12:03.601866 22530 catalog_manager.cc:1504] Loading token signing keys...
I20260924 21:12:03.608453 22530 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3277dfd00f8a463792b3f2d73ff03622: Generated new TSK 0
I20260924 21:12:03.608644 22530 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 21:12:03.610615 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAzNNjS/build/release/bin/kudu with pid 22453
2026-09-24T21:12:03Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1178 ms)
[----------] 18 tests from AuthTokenIssuingTest (21132 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790284076800039-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 21:12:03 dist-test-slave-c7tk krb5kdc[22552](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 21:12:03 dist-test-slave-c7tk krb5kdc[22552](info): set up 2 sockets
Sep 24 21:12:03 dist-test-slave-c7tk krb5kdc[22552](info): commencing operation
krb5kdc: starting...
W20260924 21:12:04.569483 392 mini_kdc.cc:121] Time spent starting KDC: real 0.940s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790284076800039-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790284076800039-392-0/krb5kdc/test-user.keytab.
Sep 24 21:12:04 dist-test-slave-c7tk krb5kdc[22552](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284324, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T21:12:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T21:12:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790284076800039-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790284076800039-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 21:12:04.676019 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAzNNjS/build/release/bin/kudu
/tmp/dist-test-taskAzNNjS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790284076800039-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790284076800039-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790284076800039-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790284076800039-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36755
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790284076800039-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41343
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36755,127.0.98.61:41801,127.0.98.60:36999
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790284076800039-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790284076800039-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790284076800039-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 21:12:04.748951 22565 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260924 21:12:04.749120 22565 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 21:12:04.749135 22565 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 21:12:04.749171 22565 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 21:12:04.750222 22565 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 21:12:04.750257 22565 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 21:12:04.750270 22565 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 21:12:04.750283 22565 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 21:12:04.750327 22565 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 21:12:04.751768 22565 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41343
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790284076800039-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790284076800039-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36755,127.0.98.61:41801,127.0.98.60:36999
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790284076800039-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36755
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790284076800039-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22565
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790284076800039-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 19:49:06 UTC on ccycloud.quasar-vhqjoc.root.comops.site
build id 4670864
I20260924 21:12:04.752020 22565 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 21:12:04.752223 22565 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 21:12:04.756615 22571 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 21:12:04.756630 22570 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 21:12:04.756670 22565 server_base.cc:1034] running on GCE node
W20260924 21:12:04.756615 22573 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 21:12:04.757074 22565 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 21:12:04.757278 22565 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 21:12:04.758401 22565 hybrid_clock.cc:648] HybridClock initialized: now 1790284324758383 us; error 33 us; skew 500 ppm
Sep 24 21:12:04 dist-test-slave-c7tk krb5kdc[22552](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284324, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 21:12:04.760481 22565 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 21:12:04.760845 22565 webserver.cc:460] Webserver started at http://127.0.98.62:44097/ using document root <none> and password file <none>
I20260924 21:12:04.761008 22565 fs_manager.cc:359] Metadata directory not provided
I20260924 21:12:04.761046 22565 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 21:12:04.761127 22565 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 21:12:04.762108 22565 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790284076800039-392-0/minicluster-data/master-0/data/instance:
uuid: "024fa4c9c5594295af50083f4f3e4289"
format_stamp: "Formatted at 2026-09-24 21:12:04 on dist-test-slave-c7tk"
server_key: "61589f4daec3b896e0b9920078a11e43"
server_key_iv: "0544dafa41b15584f3f4f236c70c5a98"
server_key_version: "encryptionkey@0"
I20260924 21:12:04.762372 22565 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790284076800039-392-0/minicluster-data/master-0/wal/instance:
uuid: "024fa4c9c5594295af50083f4f3e4289"
format_stamp: "Formatted at 2026-09-24 21:12:04 on dist-test-slave-c7tk"
server_key: "61589f4daec3b896e0b9920078a11e43"
server_key_iv: "0544dafa41b15584f3f4f236c70c5a98"
server_key_version: "encryptionkey@0"
I20260924 21:12:04.763473 22565 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 21:12:04.764111 22580 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 21:12:04.764309 22565 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 21:12:04.764420 22565 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790284076800039-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790284076800039-392-0/minicluster-data/master-0/wal
uuid: "024fa4c9c5594295af50083f4f3e4289"
format_stamp: "Formatted at 2026-09-24 21:12:04 on dist-test-slave-c7tk"
server_key: "61589f4daec3b896e0b9920078a11e43"
server_key_iv: "0544dafa41b15584f3f4f236c70c5a98"
server_key_version: "encryptionkey@0"
I20260924 21:12:04.764497 22565 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790284076800039-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790284076800039-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790284076800039-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 21:12:04.776317 22565 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 21:12:04.778299 22565 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 21:12:04.778477 22565 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 21:12:04.782639 22565 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36755
I20260924 21:12:04.782665 22632 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36755 every 8 connection(s)
I20260924 21:12:04.782986 22565 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790284076800039-392-0/minicluster-data/master-0/data/info.pb
I20260924 21:12:04.783840 22633 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 21:12:04.785043 22633 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36755 } has no permanent_uuid. Determining permanent_uuid...
I20260924 21:12:04.789935 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAzNNjS/build/release/bin/kudu as pid 22565
I20260924 21:12:04.790043 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790284076800039-392-0/minicluster-data/master-0/wal/instance
I20260924 21:12:04.790210 392 external_mini_cluster.cc:1428] Setting key 4b72b56784e992bcca93b82a528b3469
Sep 24 21:12:04 dist-test-slave-c7tk krb5kdc[22552](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284324, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 21:12:04.793789 22635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:12:04.785549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36755 (local address 127.0.0.1:59022)
0924 21:12:04.785685 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0924 21:12:04.785721 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0924 21:12:04.786132 (+ 411us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 21:12:04.786379 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 21:12:04.786381 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 21:12:04.786599 (+ 218us) client_negotiation.cc:189] Negotiated authn=SASL
0924 21:12:04.786713 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:12:04.786716 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:12:04.787600 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:12:04.787601 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:12:04.788190 (+ 589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:12:04.788192 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:12:04.788667 (+ 475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:12:04.788668 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:12:04.788741 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:12:04.789181 (+ 440us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 21:12:04.789188 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0924 21:12:04.790768 (+ 1580us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 21:12:04.792404 (+ 1636us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 21:12:04.792409 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 21:12:04.792410 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 21:12:04.792575 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 21:12:04.792855 (+ 280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 21:12:04.792856 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 21:12:04.792856 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 21:12:04.792929 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 21:12:04.793237 (+ 308us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 21:12:04.793242 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 21:12:04.793293 (+ 51us) client_negotiation.cc:769] Sending connection context
0924 21:12:04.793387 (+ 94us) client_negotiation.cc:240] Negotiation successful
0924 21:12:04.793614 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104,"mutex_wait_us":120,"thread_start_us":65,"threads_started":1}
I20260924 21:12:04.793789 22636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:12:04.785603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59022 (local address 127.0.98.62:36755)
0924 21:12:04.785735 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0924 21:12:04.785739 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 21:12:04.785821 (+ 82us) server_negotiation.cc:407] Connection header received
0924 21:12:04.786264 (+ 443us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:12:04.786266 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:12:04.786284 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:12:04.786373 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:12:04.786808 (+ 435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:12:04.787510 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:12:04.788297 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:12:04.788570 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:12:04.788679 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:12:04.790866 (+ 2187us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 21:12:04.790872 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 21:12:04.790873 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 21:12:04.790879 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 21:12:04.792337 (+ 1458us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:12:04.792700 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:12:04.792702 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:12:04.792703 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:12:04.792792 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:12:04.792990 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:12:04.792990 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:12:04.792991 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:12:04.793171 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 21:12:04.793255 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0924 21:12:04.793408 (+ 153us) server_negotiation.cc:299] Negotiation successful
0924 21:12:04.793570 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":5,"server-negotiator.queue_time_us":110,"thread_start_us":73,"threads_started":1}
I20260924 21:12:04.794392 22633 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41801 } has no permanent_uuid. Determining permanent_uuid...
I20260924 21:12:04.794682 22635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:12:04.794584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:41801 (local address 127.0.0.1:37816)
0924 21:12:04.794626 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0924 21:12:04.794661 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:41801: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260924 21:12:04.794798 22583 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:41801: connect: Connection refused (error 111)
I20260924 21:12:04.795150 22635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:12:04.795073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:41801 (local address 127.0.0.1:37828)
0924 21:12:04.795112 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0924 21:12:04.795132 (+ 20us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:41801: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260924 21:12:04.795241 22633 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:41801: Network error: Client connection negotiation failed: client connection to 127.0.98.61:41801: connect: Connection refused (error 111)
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790284076800039-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790284076800039-392-0/krb5kdc/kudu_127.0.98.61.keytab.
I20260924 21:12:04.812378 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAzNNjS/build/release/bin/kudu
/tmp/dist-test-taskAzNNjS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790284076800039-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790284076800039-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790284076800039-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790284076800039-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:41801
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790284076800039-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41343
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36755,127.0.98.61:41801,127.0.98.60:36999
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790284076800039-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790284076800039-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790284076800039-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 21:12:04.844548 22633 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41801 } attempt: 1
I20260924 21:12:04.844926 22635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:12:04.844787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:41801 (local address 127.0.0.1:37844)
0924 21:12:04.844838 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0924 21:12:04.844895 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:41801: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260924 21:12:04.845256 22635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:12:04.845167 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:41801 (local address 127.0.0.1:37854)
0924 21:12:04.845210 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0924 21:12:04.845238 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:41801: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260924 21:12:04.845361 22633 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:41801: Network error: Client connection negotiation failed: client connection to 127.0.98.61:41801: connect: Connection refused (error 111)
W20260924 21:12:04.887974 22640 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260924 21:12:04.888157 22640 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 21:12:04.888175 22640 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 21:12:04.888193 22640 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 21:12:04.889148 22640 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 21:12:04.889181 22640 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 21:12:04.889196 22640 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 21:12:04.889209 22640 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 21:12:04.889222 22640 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 21:12:04.890481 22640 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41343
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790284076800039-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790284076800039-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36755,127.0.98.61:41801,127.0.98.60:36999
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790284076800039-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:41801
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790284076800039-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22640
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790284076800039-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 19:49:06 UTC on ccycloud.quasar-vhqjoc.root.comops.site
build id 4670864
I20260924 21:12:04.890645 22640 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 21:12:04.890813 22640 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 21:12:04.895272 22646 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 21:12:04.895306 22645 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 21:12:04.895282 22648 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 21:12:04.895460 22640 server_base.cc:1034] running on GCE node
I20260924 21:12:04.895668 22640 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 21:12:04.895915 22640 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 21:12:04.897090 22640 hybrid_clock.cc:648] HybridClock initialized: now 1790284324897067 us; error 34 us; skew 500 ppm
Sep 24 21:12:04 dist-test-slave-c7tk krb5kdc[22552](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284324, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 21:12:04.899252 22640 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260924 21:12:04.899649 22640 webserver.cc:460] Webserver started at http://127.0.98.61:42597/ using document root <none> and password file <none>
I20260924 21:12:04.899849 22640 fs_manager.cc:359] Metadata directory not provided
I20260924 21:12:04.899888 22640 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 21:12:04.900017 22640 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 21:12:04.900900 22640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790284076800039-392-0/minicluster-data/master-1/data/instance:
uuid: "04b4ac56836d40a7926274191e6e3744"
format_stamp: "Formatted at 2026-09-24 21:12:04 on dist-test-slave-c7tk"
server_key: "3e169c3b9c15e3f569559cc6cd86467e"
server_key_iv: "b0d87914b4164fe1034d941d293ce9d4"
server_key_version: "encryptionkey@0"
I20260924 21:12:04.901230 22640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790284076800039-392-0/minicluster-data/master-1/wal/instance:
uuid: "04b4ac56836d40a7926274191e6e3744"
format_stamp: "Formatted at 2026-09-24 21:12:04 on dist-test-slave-c7tk"
server_key: "3e169c3b9c15e3f569559cc6cd86467e"
server_key_iv: "b0d87914b4164fe1034d941d293ce9d4"
server_key_version: "encryptionkey@0"
I20260924 21:12:04.902504 22640 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 21:12:04.903123 22655 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 21:12:04.903278 22640 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 21:12:04.903338 22640 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790284076800039-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790284076800039-392-0/minicluster-data/master-1/wal
uuid: "04b4ac56836d40a7926274191e6e3744"
format_stamp: "Formatted at 2026-09-24 21:12:04 on dist-test-slave-c7tk"
server_key: "3e169c3b9c15e3f569559cc6cd86467e"
server_key_iv: "b0d87914b4164fe1034d941d293ce9d4"
server_key_version: "encryptionkey@0"
I20260924 21:12:04.903411 22640 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790284076800039-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790284076800039-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790284076800039-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 21:12:04.913509 22633 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41801 } attempt: 2
I20260924 21:12:04.913868 22635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:12:04.913729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:41801 (local address 127.0.0.1:37856)
0924 21:12:04.913780 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0924 21:12:04.913846 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:41801: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260924 21:12:04.914125 22635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:12:04.914047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:41801 (local address 127.0.0.1:37862)
0924 21:12:04.914084 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0924 21:12:04.914104 (+ 20us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:41801: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
W20260924 21:12:04.914199 22633 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:41801: Network error: Client connection negotiation failed: client connection to 127.0.98.61:41801: connect: Connection refused (error 111)
I20260924 21:12:04.919960 22640 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 21:12:04.922065 22640 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 21:12:04.922191 22640 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 21:12:04.925572 22640 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:41801
I20260924 21:12:04.925601 22707 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:41801 every 8 connection(s)
I20260924 21:12:04.925963 22640 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790284076800039-392-0/minicluster-data/master-1/data/info.pb
I20260924 21:12:04.926445 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAzNNjS/build/release/bin/kudu as pid 22640
I20260924 21:12:04.926543 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790284076800039-392-0/minicluster-data/master-1/wal/instance
I20260924 21:12:04.926735 392 external_mini_cluster.cc:1428] Setting key 143cb611b63fc9df437fb6ece7ac6c54
I20260924 21:12:04.927011 22708 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 21:12:04.928656 22708 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36755 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
Sep 24 21:12:04 dist-test-slave-c7tk krb5kdc[22552](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284324, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 21:12:04.939333 22636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:12:04.929252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59026 (local address 127.0.98.62:36755)
0924 21:12:04.929305 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0924 21:12:04.929307 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 21:12:04.929453 (+ 146us) server_negotiation.cc:407] Connection header received
0924 21:12:04.929964 (+ 511us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:12:04.929966 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:12:04.929979 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:12:04.930055 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:12:04.930512 (+ 457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:12:04.931336 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:12:04.932269 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:12:04.932554 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:12:04.932653 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:12:04.936822 (+ 4169us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 21:12:04.936824 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 21:12:04.936824 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 21:12:04.936830 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 21:12:04.938066 (+ 1236us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:12:04.938429 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:12:04.938430 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:12:04.938431 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:12:04.938485 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:12:04.938724 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:12:04.938724 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:12:04.938725 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:12:04.938827 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 21:12:04.938913 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0924 21:12:04.939176 (+ 263us) server_negotiation.cc:299] Negotiation successful
0924 21:12:04.939252 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260924 21:12:04.939498 22711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:12:04.929205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36755 (local address 127.0.0.1:59026)
0924 21:12:04.929340 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0924 21:12:04.929373 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0924 21:12:04.929875 (+ 502us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 21:12:04.930096 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 21:12:04.930099 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 21:12:04.930285 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0924 21:12:04.930430 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:12:04.930433 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:12:04.931437 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:12:04.931437 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:12:04.932183 (+ 746us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:12:04.932185 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:12:04.932673 (+ 488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:12:04.932674 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:12:04.932759 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:12:04.933182 (+ 423us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 21:12:04.933188 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0924 21:12:04.936734 (+ 3546us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 21:12:04.938140 (+ 1406us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 21:12:04.938146 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 21:12:04.938148 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 21:12:04.938345 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 21:12:04.938562 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 21:12:04.938563 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 21:12:04.938563 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 21:12:04.938662 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 21:12:04.938918 (+ 256us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 21:12:04.938924 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 21:12:04.939053 (+ 129us) client_negotiation.cc:769] Sending connection context
0924 21:12:04.939156 (+ 103us) client_negotiation.cc:240] Negotiation successful
0924 21:12:04.939290 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":74,"threads_started":1}
I20260924 21:12:04.940033 22708 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41801 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 21:12:04 dist-test-slave-c7tk krb5kdc[22552](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284324, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260924 21:12:04.947264 22711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:12:04.940235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:41801 (local address 127.0.0.1:37870)
0924 21:12:04.940277 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0924 21:12:04.940286 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 21:12:04.940336 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 21:12:04.940555 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 21:12:04.940555 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 21:12:04.940676 (+ 121us) client_negotiation.cc:189] Negotiated authn=SASL
0924 21:12:04.940711 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:12:04.940713 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:12:04.941545 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:12:04.941546 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:12:04.942111 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:12:04.942113 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:12:04.942550 (+ 437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:12:04.942551 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:12:04.942608 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:12:04.942885 (+ 277us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 21:12:04.942889 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 21:12:04.944765 (+ 1876us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 21:12:04.946445 (+ 1680us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 21:12:04.946445 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 21:12:04.946446 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 21:12:04.946591 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 21:12:04.946760 (+ 169us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 21:12:04.946761 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 21:12:04.946761 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 21:12:04.946805 (+ 44us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 21:12:04.947069 (+ 264us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 21:12:04.947070 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 21:12:04.947102 (+ 32us) client_negotiation.cc:769] Sending connection context
0924 21:12:04.947150 (+ 48us) client_negotiation.cc:240] Negotiation successful
0924 21:12:04.947215 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28}
I20260924 21:12:04.947414 22713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:12:04.940295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37870 (local address 127.0.98.61:41801)
0924 21:12:04.940440 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0924 21:12:04.940444 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 21:12:04.940457 (+ 13us) server_negotiation.cc:407] Connection header received
0924 21:12:04.940471 (+ 14us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:12:04.940472 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:12:04.940481 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:12:04.940554 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:12:04.940805 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:12:04.941459 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:12:04.942193 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:12:04.942466 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:12:04.942550 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:12:04.944843 (+ 2293us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 21:12:04.944849 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 21:12:04.944849 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 21:12:04.944854 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 21:12:04.946375 (+ 1521us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:12:04.946655 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:12:04.946657 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:12:04.946657 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:12:04.946702 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:12:04.946856 (+ 154us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:12:04.946857 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:12:04.946857 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:12:04.947009 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 21:12:04.947072 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0924 21:12:04.947169 (+ 97us) server_negotiation.cc:299] Negotiation successful
0924 21:12:04.947256 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":33,"server-negotiator.queue_time_us":116,"thread_start_us":78,"threads_started":1}
I20260924 21:12:04.947996 22708 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36999 } has no permanent_uuid. Determining permanent_uuid...
I20260924 21:12:04.948287 22711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:12:04.948187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36999 (local address 127.0.0.1:58466)
0924 21:12:04.948229 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0924 21:12:04.948265 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36999: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260924 21:12:04.948484 22658 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36999: connect: Connection refused (error 111)
I20260924 21:12:04.948827 22711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:12:04.948751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36999 (local address 127.0.0.1:58480)
0924 21:12:04.948793 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0924 21:12:04.948814 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36999: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260924 21:12:04.948987 22708 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:36999: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36999: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790284076800039-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790284076800039-392-0/krb5kdc/kudu_127.0.98.60.keytab.
I20260924 21:12:04.950103 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAzNNjS/build/release/bin/kudu
/tmp/dist-test-taskAzNNjS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790284076800039-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790284076800039-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790284076800039-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790284076800039-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:36999
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790284076800039-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41343
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36755,127.0.98.61:41801,127.0.98.60:36999
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790284076800039-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790284076800039-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790284076800039-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 21:12:04.998211 22708 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36999 } attempt: 1
I20260924 21:12:04.998761 22711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:12:04.998619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36999 (local address 127.0.0.1:58492)
0924 21:12:04.998701 (+ 82us) negotiation.cc:107] Waiting for socket to connect
0924 21:12:04.998736 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36999: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260924 21:12:04.999148 22711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:12:04.999059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36999 (local address 127.0.0.1:58496)
0924 21:12:04.999109 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0924 21:12:04.999132 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36999: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260924 21:12:04.999245 22708 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:36999: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36999: connect: Connection refused (error 111)
I20260924 21:12:05.005429 22633 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41801 } attempt: 3
Sep 24 21:12:05 dist-test-slave-c7tk krb5kdc[22552](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284324, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260924 21:12:05.011708 22713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:12:05.005728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37880 (local address 127.0.98.61:41801)
0924 21:12:05.005766 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0924 21:12:05.005768 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 21:12:05.005776 (+ 8us) server_negotiation.cc:407] Connection header received
0924 21:12:05.005858 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:12:05.005859 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:12:05.005868 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:12:05.005931 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:12:05.006203 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:12:05.006676 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:12:05.007306 (+ 630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:12:05.007494 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:12:05.007564 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:12:05.009334 (+ 1770us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 21:12:05.009336 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 21:12:05.009336 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 21:12:05.009343 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 21:12:05.010611 (+ 1268us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:12:05.010951 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:12:05.010952 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:12:05.010953 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:12:05.011012 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:12:05.011208 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:12:05.011209 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:12:05.011209 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:12:05.011317 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 21:12:05.011389 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0924 21:12:05.011552 (+ 163us) server_negotiation.cc:299] Negotiation successful
0924 21:12:05.011622 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260924 21:12:05.011713 22635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:12:05.005683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:41801 (local address 127.0.0.1:37880)
0924 21:12:05.005734 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0924 21:12:05.005744 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 21:12:05.005788 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 21:12:05.005935 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 21:12:05.005935 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 21:12:05.006079 (+ 144us) client_negotiation.cc:189] Negotiated authn=SASL
0924 21:12:05.006131 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:12:05.006133 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:12:05.006758 (+ 625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:12:05.006759 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:12:05.007218 (+ 459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:12:05.007220 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:12:05.007574 (+ 354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:12:05.007574 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:12:05.007646 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:12:05.008062 (+ 416us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 21:12:05.008066 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 21:12:05.009256 (+ 1190us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 21:12:05.010687 (+ 1431us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 21:12:05.010688 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 21:12:05.010688 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 21:12:05.010876 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 21:12:05.011081 (+ 205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 21:12:05.011082 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 21:12:05.011082 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 21:12:05.011144 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 21:12:05.011395 (+ 251us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 21:12:05.011396 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 21:12:05.011439 (+ 43us) client_negotiation.cc:769] Sending connection context
0924 21:12:05.011528 (+ 89us) client_negotiation.cc:240] Negotiation successful
0924 21:12:05.011622 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260924 21:12:05.012147 22633 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36999 } has no permanent_uuid. Determining permanent_uuid...
I20260924 21:12:05.012499 22635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:12:05.012403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36999 (local address 127.0.0.1:58500)
0924 21:12:05.012459 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0924 21:12:05.012483 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36999: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260924 21:12:05.012775 22635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:12:05.012698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36999 (local address 127.0.0.1:58514)
0924 21:12:05.012737 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0924 21:12:05.012757 (+ 20us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36999: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260924 21:12:05.012857 22633 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:36999: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36999: connect: Connection refused (error 111)
W20260924 21:12:05.021888 22715 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260924 21:12:05.022070 22715 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 21:12:05.022099 22715 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 21:12:05.022152 22715 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 21:12:05.023096 22715 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 21:12:05.023129 22715 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 21:12:05.023140 22715 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 21:12:05.023185 22715 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 21:12:05.023214 22715 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 21:12:05.024434 22715 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41343
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790284076800039-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790284076800039-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36755,127.0.98.61:41801,127.0.98.60:36999
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790284076800039-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:36999
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790284076800039-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22715
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790284076800039-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 19:49:06 UTC on ccycloud.quasar-vhqjoc.root.comops.site
build id 4670864
I20260924 21:12:05.024602 22715 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 21:12:05.024791 22715 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 21:12:05.028818 22721 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 21:12:05.028867 22715 server_base.cc:1034] running on GCE node
W20260924 21:12:05.028839 22720 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 21:12:05.028873 22723 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 21:12:05.029325 22715 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 21:12:05.029615 22715 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 21:12:05.030758 22715 hybrid_clock.cc:648] HybridClock initialized: now 1790284325030740 us; error 27 us; skew 500 ppm
Sep 24 21:12:05 dist-test-slave-c7tk krb5kdc[22552](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284325, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 21:12:05.032829 22715 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260924 21:12:05.033196 22715 webserver.cc:460] Webserver started at http://127.0.98.60:45915/ using document root <none> and password file <none>
I20260924 21:12:05.033420 22715 fs_manager.cc:359] Metadata directory not provided
I20260924 21:12:05.033480 22715 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 21:12:05.033624 22715 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 21:12:05.034530 22715 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790284076800039-392-0/minicluster-data/master-2/data/instance:
uuid: "79a0407db57b41b8a3813815f5a8cfc6"
format_stamp: "Formatted at 2026-09-24 21:12:05 on dist-test-slave-c7tk"
server_key: "0bc3f69309cf38f1e69fdd9e049c7117"
server_key_iv: "3d6f843e53fd4a28b489db946928aa92"
server_key_version: "encryptionkey@0"
I20260924 21:12:05.034849 22715 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790284076800039-392-0/minicluster-data/master-2/wal/instance:
uuid: "79a0407db57b41b8a3813815f5a8cfc6"
format_stamp: "Formatted at 2026-09-24 21:12:05 on dist-test-slave-c7tk"
server_key: "0bc3f69309cf38f1e69fdd9e049c7117"
server_key_iv: "3d6f843e53fd4a28b489db946928aa92"
server_key_version: "encryptionkey@0"
I20260924 21:12:05.035959 22715 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 21:12:05.036535 22730 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 21:12:05.036700 22715 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 21:12:05.036787 22715 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790284076800039-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790284076800039-392-0/minicluster-data/master-2/wal
uuid: "79a0407db57b41b8a3813815f5a8cfc6"
format_stamp: "Formatted at 2026-09-24 21:12:05 on dist-test-slave-c7tk"
server_key: "0bc3f69309cf38f1e69fdd9e049c7117"
server_key_iv: "3d6f843e53fd4a28b489db946928aa92"
server_key_version: "encryptionkey@0"
I20260924 21:12:05.036876 22715 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790284076800039-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790284076800039-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790284076800039-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 21:12:05.043980 22633 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36999 } attempt: 1
I20260924 21:12:05.044296 22635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:12:05.044197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36999 (local address 127.0.0.1:58518)
0924 21:12:05.044243 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0924 21:12:05.044274 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36999: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260924 21:12:05.044566 22635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:12:05.044498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36999 (local address 127.0.0.1:58526)
0924 21:12:05.044532 (+ 34us) negotiation.cc:107] Waiting for socket to connect
0924 21:12:05.044553 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36999: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
W20260924 21:12:05.044675 22633 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:36999: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36999: connect: Connection refused (error 111)
I20260924 21:12:05.051421 22715 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 21:12:05.053056 22715 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 21:12:05.053213 22715 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 21:12:05.056679 22715 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:36999
I20260924 21:12:05.056734 22782 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:36999 every 8 connection(s)
I20260924 21:12:05.057039 22715 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790284076800039-392-0/minicluster-data/master-2/data/info.pb
I20260924 21:12:05.057964 22783 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 21:12:05.059181 22783 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36755 } has no permanent_uuid. Determining permanent_uuid...
I20260924 21:12:05.063760 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAzNNjS/build/release/bin/kudu as pid 22715
Sep 24 21:12:05 dist-test-slave-c7tk krb5kdc[22552](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284325, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 21:12:05.063845 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790284076800039-392-0/minicluster-data/master-2/wal/instance
I20260924 21:12:05.064028 392 external_mini_cluster.cc:1428] Setting key 21e9dcb923e512dbccb5f7b42eb65b3d
I20260924 21:12:05.066926 22636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:12:05.059736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59036 (local address 127.0.98.62:36755)
0924 21:12:05.059813 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0924 21:12:05.059815 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 21:12:05.059943 (+ 128us) server_negotiation.cc:407] Connection header received
0924 21:12:05.060299 (+ 356us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:12:05.060299 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:12:05.060308 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:12:05.060353 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:12:05.060851 (+ 498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:12:05.061298 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:12:05.062257 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:12:05.062411 (+ 154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:12:05.062460 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:12:05.064273 (+ 1813us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 21:12:05.064274 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 21:12:05.064275 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 21:12:05.064280 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 21:12:05.065725 (+ 1445us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:12:05.066097 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:12:05.066098 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:12:05.066098 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:12:05.066167 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:12:05.066381 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:12:05.066383 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:12:05.066383 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:12:05.066505 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 21:12:05.066564 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0924 21:12:05.066778 (+ 214us) server_negotiation.cc:299] Negotiation successful
0924 21:12:05.066841 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260924 21:12:05.067088 22785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:12:05.059703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36755 (local address 127.0.0.1:59036)
0924 21:12:05.059845 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0924 21:12:05.059872 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0924 21:12:05.060233 (+ 361us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 21:12:05.060396 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 21:12:05.060398 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 21:12:05.060600 (+ 202us) client_negotiation.cc:189] Negotiated authn=SASL
0924 21:12:05.060757 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:12:05.060766 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:12:05.061368 (+ 602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:12:05.061369 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:12:05.062182 (+ 813us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:12:05.062184 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:12:05.062486 (+ 302us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:12:05.062487 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:12:05.062567 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:12:05.062969 (+ 402us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 21:12:05.062981 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 21:12:05.064183 (+ 1202us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 21:12:05.065831 (+ 1648us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 21:12:05.065836 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 21:12:05.065838 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 21:12:05.066033 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 21:12:05.066225 (+ 192us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 21:12:05.066225 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 21:12:05.066226 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 21:12:05.066306 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 21:12:05.066568 (+ 262us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 21:12:05.066574 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 21:12:05.066671 (+ 97us) client_negotiation.cc:769] Sending connection context
0924 21:12:05.066760 (+ 89us) client_negotiation.cc:240] Negotiation successful
0924 21:12:05.066898 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":84,"threads_started":1}
I20260924 21:12:05.067373 22708 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36999 } attempt: 2
I20260924 21:12:05.067777 22783 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41801 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 21:12:05 dist-test-slave-c7tk krb5kdc[22552](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284324, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Sep 24 21:12:05 dist-test-slave-c7tk krb5kdc[22552](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284324, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Sep 24 21:12:05 dist-test-slave-c7tk krb5kdc[22552](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284325, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260924 21:12:05.074005 22787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:12:05.064353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58542 (local address 127.0.98.60:36999)
0924 21:12:05.064521 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0924 21:12:05.064525 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 21:12:05.064610 (+ 85us) server_negotiation.cc:407] Connection header received
0924 21:12:05.064634 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:12:05.064635 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:12:05.064645 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:12:05.064729 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:12:05.065356 (+ 627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:12:05.066276 (+ 920us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:12:05.066947 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:12:05.067367 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:12:05.067446 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:12:05.069878 (+ 2432us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 21:12:05.069884 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 21:12:05.069884 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 21:12:05.069890 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 21:12:05.072024 (+ 2134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:12:05.072749 (+ 725us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:12:05.072751 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:12:05.072751 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:12:05.073180 (+ 429us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:12:05.073390 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:12:05.073391 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:12:05.073391 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:12:05.073561 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 21:12:05.073711 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0924 21:12:05.073798 (+ 87us) server_negotiation.cc:299] Negotiation successful
0924 21:12:05.073897 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":70,"threads_started":1}
I20260924 21:12:05.074631 22711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:12:05.067641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36999 (local address 127.0.0.1:58546)
0924 21:12:05.067687 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0924 21:12:05.067696 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 21:12:05.067728 (+ 32us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 21:12:05.068087 (+ 359us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 21:12:05.068089 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 21:12:05.068249 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0924 21:12:05.068290 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:12:05.068293 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:12:05.069005 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:12:05.069006 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:12:05.069702 (+ 696us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:12:05.069704 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:12:05.070115 (+ 411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:12:05.070116 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:12:05.070163 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:12:05.070469 (+ 306us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 21:12:05.070472 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0924 21:12:05.071752 (+ 1280us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 21:12:05.073368 (+ 1616us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 21:12:05.073369 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 21:12:05.073369 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 21:12:05.073576 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 21:12:05.074028 (+ 452us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 21:12:05.074029 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 21:12:05.074029 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 21:12:05.074088 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 21:12:05.074320 (+ 232us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 21:12:05.074321 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 21:12:05.074356 (+ 35us) client_negotiation.cc:769] Sending connection context
0924 21:12:05.074466 (+ 110us) client_negotiation.cc:240] Negotiation successful
0924 21:12:05.074551 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260924 21:12:05.074676 22788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:12:05.067669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58546 (local address 127.0.98.60:36999)
0924 21:12:05.067968 (+ 299us) server_negotiation.cc:206] Beginning negotiation
0924 21:12:05.067969 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 21:12:05.067976 (+ 7us) server_negotiation.cc:407] Connection header received
0924 21:12:05.067994 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:12:05.067994 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:12:05.068012 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:12:05.068077 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:12:05.068343 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:12:05.068939 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:12:05.069784 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:12:05.070043 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:12:05.070104 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:12:05.071836 (+ 1732us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 21:12:05.071838 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 21:12:05.071838 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 21:12:05.071845 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 21:12:05.073295 (+ 1450us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:12:05.073901 (+ 606us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:12:05.073902 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:12:05.073903 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:12:05.073957 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:12:05.074153 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:12:05.074154 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:12:05.074154 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:12:05.074253 (+ 99us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 21:12:05.074315 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0924 21:12:05.074488 (+ 173us) server_negotiation.cc:299] Negotiation successful
0924 21:12:05.074580 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":280,"thread_start_us":68,"threads_started":1}
I20260924 21:12:05.075394 22785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:12:05.068022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:41801 (local address 127.0.0.1:37894)
0924 21:12:05.068140 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0924 21:12:05.068149 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 21:12:05.068192 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 21:12:05.068299 (+ 107us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 21:12:05.068300 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 21:12:05.068438 (+ 138us) client_negotiation.cc:189] Negotiated authn=SASL
0924 21:12:05.068491 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:12:05.068493 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:12:05.069216 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:12:05.069216 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:12:05.069690 (+ 474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:12:05.069692 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:12:05.071013 (+ 1321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:12:05.071014 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:12:05.071081 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:12:05.071453 (+ 372us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 21:12:05.071459 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0924 21:12:05.072745 (+ 1286us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 21:12:05.074384 (+ 1639us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 21:12:05.074385 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 21:12:05.074385 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 21:12:05.074607 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 21:12:05.074793 (+ 186us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 21:12:05.074794 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 21:12:05.074794 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 21:12:05.074866 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 21:12:05.075122 (+ 256us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 21:12:05.075123 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 21:12:05.075157 (+ 34us) client_negotiation.cc:769] Sending connection context
0924 21:12:05.075212 (+ 55us) client_negotiation.cc:240] Negotiation successful
0924 21:12:05.075289 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105}
I20260924 21:12:05.075675 22713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:12:05.068103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37894 (local address 127.0.98.61:41801)
0924 21:12:05.068147 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0924 21:12:05.068148 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 21:12:05.068180 (+ 32us) server_negotiation.cc:407] Connection header received
0924 21:12:05.068242 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:12:05.068243 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:12:05.068251 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:12:05.068294 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:12:05.068567 (+ 273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:12:05.069141 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:12:05.069759 (+ 618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:12:05.070923 (+ 1164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:12:05.071003 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:12:05.072833 (+ 1830us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 21:12:05.072834 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 21:12:05.072835 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 21:12:05.072841 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 21:12:05.074301 (+ 1460us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:12:05.074679 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:12:05.074680 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:12:05.074681 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:12:05.074732 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:12:05.074929 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:12:05.074930 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:12:05.074930 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:12:05.075044 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 21:12:05.075097 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0924 21:12:05.075527 (+ 430us) server_negotiation.cc:299] Negotiation successful
0924 21:12:05.075594 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260924 21:12:05.075934 22708 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 04b4ac56836d40a7926274191e6e3744: Bootstrap starting.
I20260924 21:12:05.076562 22708 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 04b4ac56836d40a7926274191e6e3744: Neither blocks nor log segments found. Creating new log.
I20260924 21:12:05.076814 22708 log.cc:824] T 00000000000000000000000000000000 P 04b4ac56836d40a7926274191e6e3744: Log is configured to *not* fsync() on all Append() calls
I20260924 21:12:05.077243 22783 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36999 } has no permanent_uuid. Determining permanent_uuid...
I20260924 21:12:05.077485 22708 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 04b4ac56836d40a7926274191e6e3744: No bootstrap required, opened a new log
Sep 24 21:12:05 dist-test-slave-c7tk krb5kdc[22552](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284324, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 21:12:05.079373 22708 raft_consensus.cc:348] T 00000000000000000000000000000000 P 04b4ac56836d40a7926274191e6e3744 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "024fa4c9c5594295af50083f4f3e4289" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36755 } } peers { permanent_uuid: "04b4ac56836d40a7926274191e6e3744" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41801 } } peers { permanent_uuid: "79a0407db57b41b8a3813815f5a8cfc6" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36999 } }
I20260924 21:12:05.079509 22708 raft_consensus.cc:729] T 00000000000000000000000000000000 P 04b4ac56836d40a7926274191e6e3744 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 04b4ac56836d40a7926274191e6e3744, State: Initialized, Role: FOLLOWER
I20260924 21:12:05.079577 22708 consensus_queue.cc:260] T 00000000000000000000000000000000 P 04b4ac56836d40a7926274191e6e3744 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "024fa4c9c5594295af50083f4f3e4289" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36755 } } peers { permanent_uuid: "04b4ac56836d40a7926274191e6e3744" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41801 } } peers { permanent_uuid: "79a0407db57b41b8a3813815f5a8cfc6" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36999 } }
I20260924 21:12:05.080158 22708 sys_catalog.cc:564] T 00000000000000000000000000000000 P 04b4ac56836d40a7926274191e6e3744 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 21:12:05.080176 22790 sys_catalog.cc:455] T 00000000000000000000000000000000 P 04b4ac56836d40a7926274191e6e3744 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "024fa4c9c5594295af50083f4f3e4289" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36755 } } peers { permanent_uuid: "04b4ac56836d40a7926274191e6e3744" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41801 } } peers { permanent_uuid: "79a0407db57b41b8a3813815f5a8cfc6" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36999 } } }
I20260924 21:12:05.080286 22790 sys_catalog.cc:458] T 00000000000000000000000000000000 P 04b4ac56836d40a7926274191e6e3744 [sys.catalog]: This master's current role is: FOLLOWER
Sep 24 21:12:05 dist-test-slave-c7tk krb5kdc[22552](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284325, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
W20260924 21:12:05.082508 22801 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 04b4ac56836d40a7926274191e6e3744: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 21:12:05.082569 22801 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 21:12:05.082576 22640 master_runner.cc:186] Error getting master registration for 127.0.98.62:36755: OK, instance_id { permanent_uuid: "024fa4c9c5594295af50083f4f3e4289" instance_seqno: 1790284324781482 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260924 21:12:05.084959 22640 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260924 21:12:05.085134 22636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:12:05.074946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59048 (local address 127.0.98.62:36755)
0924 21:12:05.074975 (+ 29us) server_negotiation.cc:206] Beginning negotiation
0924 21:12:05.074976 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 21:12:05.075054 (+ 78us) server_negotiation.cc:407] Connection header received
0924 21:12:05.075137 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:12:05.075138 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:12:05.075148 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:12:05.075190 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:12:05.075708 (+ 518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:12:05.076303 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:12:05.076964 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:12:05.077214 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:12:05.077269 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:12:05.079221 (+ 1952us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 21:12:05.079222 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 21:12:05.079222 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 21:12:05.079228 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 21:12:05.083765 (+ 4537us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:12:05.084127 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:12:05.084128 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:12:05.084128 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:12:05.084178 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:12:05.084356 (+ 178us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:12:05.084357 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:12:05.084357 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:12:05.084465 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 21:12:05.084586 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0924 21:12:05.084969 (+ 383us) server_negotiation.cc:299] Negotiation successful
0924 21:12:05.085052 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260924 21:12:05.085207 22785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:12:05.077432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36999 (local address 127.0.0.1:58556)
0924 21:12:05.077485 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0924 21:12:05.077493 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0924 21:12:05.077535 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 21:12:05.077655 (+ 120us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 21:12:05.077656 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 21:12:05.077791 (+ 135us) client_negotiation.cc:189] Negotiated authn=SASL
0924 21:12:05.077852 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:12:05.077854 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:12:05.078636 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:12:05.078637 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:12:05.079277 (+ 640us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:12:05.079279 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:12:05.079702 (+ 423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:12:05.079702 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:12:05.079753 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:12:05.080151 (+ 398us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 21:12:05.080156 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 21:12:05.081835 (+ 1679us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 21:12:05.084320 (+ 2485us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 21:12:05.084320 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 21:12:05.084321 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 21:12:05.084494 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 21:12:05.084666 (+ 172us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 21:12:05.084667 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 21:12:05.084667 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 21:12:05.084715 (+ 48us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 21:12:05.084966 (+ 251us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 21:12:05.084967 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 21:12:05.084997 (+ 30us) client_negotiation.cc:769] Sending connection context
0924 21:12:05.085052 (+ 55us) client_negotiation.cc:240] Negotiation successful
0924 21:12:05.085119 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42}
I20260924 21:12:05.085417 22788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:12:05.077457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58556 (local address 127.0.98.60:36999)
0924 21:12:05.077487 (+ 30us) server_negotiation.cc:206] Beginning negotiation
0924 21:12:05.077489 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 21:12:05.077523 (+ 34us) server_negotiation.cc:407] Connection header received
0924 21:12:05.077596 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:12:05.077597 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:12:05.077607 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:12:05.077647 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:12:05.077919 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:12:05.078557 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:12:05.079359 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:12:05.079606 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:12:05.079867 (+ 261us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:12:05.081929 (+ 2062us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 21:12:05.081930 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 21:12:05.081931 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 21:12:05.081938 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 21:12:05.084254 (+ 2316us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:12:05.084566 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:12:05.084566 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:12:05.084567 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:12:05.084612 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:12:05.084767 (+ 155us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:12:05.084767 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:12:05.084767 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:12:05.084896 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 21:12:05.084948 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0924 21:12:05.085288 (+ 340us) server_negotiation.cc:299] Negotiation successful
0924 21:12:05.085351 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260924 21:12:05.086683 22783 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 79a0407db57b41b8a3813815f5a8cfc6: Bootstrap starting.
I20260924 21:12:05.087242 22783 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 79a0407db57b41b8a3813815f5a8cfc6: Neither blocks nor log segments found. Creating new log.
I20260924 21:12:05.087505 22783 log.cc:824] T 00000000000000000000000000000000 P 79a0407db57b41b8a3813815f5a8cfc6: Log is configured to *not* fsync() on all Append() calls
I20260924 21:12:05.088263 22783 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 79a0407db57b41b8a3813815f5a8cfc6: No bootstrap required, opened a new log
Sep 24 21:12:05 dist-test-slave-c7tk krb5kdc[22552](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284324, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260924 21:12:05.090072 22783 raft_consensus.cc:348] T 00000000000000000000000000000000 P 79a0407db57b41b8a3813815f5a8cfc6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "024fa4c9c5594295af50083f4f3e4289" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36755 } } peers { permanent_uuid: "04b4ac56836d40a7926274191e6e3744" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41801 } } peers { permanent_uuid: "79a0407db57b41b8a3813815f5a8cfc6" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36999 } }
I20260924 21:12:05.090209 22783 raft_consensus.cc:729] T 00000000000000000000000000000000 P 79a0407db57b41b8a3813815f5a8cfc6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 79a0407db57b41b8a3813815f5a8cfc6, State: Initialized, Role: FOLLOWER
I20260924 21:12:05.090291 22783 consensus_queue.cc:260] T 00000000000000000000000000000000 P 79a0407db57b41b8a3813815f5a8cfc6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "024fa4c9c5594295af50083f4f3e4289" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36755 } } peers { permanent_uuid: "04b4ac56836d40a7926274191e6e3744" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41801 } } peers { permanent_uuid: "79a0407db57b41b8a3813815f5a8cfc6" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36999 } }
I20260924 21:12:05.090559 22783 sys_catalog.cc:564] T 00000000000000000000000000000000 P 79a0407db57b41b8a3813815f5a8cfc6 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 21:12:05.090538 22804 sys_catalog.cc:455] T 00000000000000000000000000000000 P 79a0407db57b41b8a3813815f5a8cfc6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "024fa4c9c5594295af50083f4f3e4289" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36755 } } peers { permanent_uuid: "04b4ac56836d40a7926274191e6e3744" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41801 } } peers { permanent_uuid: "79a0407db57b41b8a3813815f5a8cfc6" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36999 } } }
I20260924 21:12:05.090728 22804 sys_catalog.cc:458] T 00000000000000000000000000000000 P 79a0407db57b41b8a3813815f5a8cfc6 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 21:12:05.092808 22713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:12:05.085779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37898 (local address 127.0.98.61:41801)
0924 21:12:05.085827 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0924 21:12:05.085829 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 21:12:05.085869 (+ 40us) server_negotiation.cc:407] Connection header received
0924 21:12:05.085935 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:12:05.085936 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:12:05.085945 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:12:05.085996 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:12:05.086468 (+ 472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:12:05.086876 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:12:05.087418 (+ 542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:12:05.087588 (+ 170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:12:05.087650 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:12:05.089832 (+ 2182us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 21:12:05.089833 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 21:12:05.089834 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 21:12:05.089840 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 21:12:05.091474 (+ 1634us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:12:05.091822 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:12:05.091823 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:12:05.091824 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:12:05.091882 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:12:05.092184 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:12:05.092185 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:12:05.092186 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:12:05.092316 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 21:12:05.092420 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0924 21:12:05.092657 (+ 237us) server_negotiation.cc:299] Negotiation successful
0924 21:12:05.092729 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
W20260924 21:12:05.094434 22815 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 79a0407db57b41b8a3813815f5a8cfc6: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 21:12:05.094484 22815 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 21:12:05.092700 22715 master_runner.cc:186] Error getting master registration for 127.0.98.62:36755: OK, instance_id { permanent_uuid: "024fa4c9c5594295af50083f4f3e4289" instance_seqno: 1790284324781482 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260924 21:12:05.094595 22715 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260924 21:12:05.095148 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 21:12:05.095978 22607 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "024fa4c9c5594295af50083f4f3e4289"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59048
I20260924 21:12:05.119814 22633 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36999 } attempt: 2
Sep 24 21:12:05 dist-test-slave-c7tk krb5kdc[22552](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284324, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260924 21:12:05.126816 22788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:12:05.120093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58558 (local address 127.0.98.60:36999)
0924 21:12:05.120133 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0924 21:12:05.120134 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 21:12:05.120141 (+ 7us) server_negotiation.cc:407] Connection header received
0924 21:12:05.120160 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:12:05.120161 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:12:05.120169 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:12:05.120221 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:12:05.120499 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:12:05.121213 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:12:05.122179 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:12:05.122437 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:12:05.122502 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:12:05.124248 (+ 1746us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 21:12:05.124250 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 21:12:05.124250 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 21:12:05.124256 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 21:12:05.125775 (+ 1519us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:12:05.126153 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:12:05.126154 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:12:05.126154 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:12:05.126219 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:12:05.126394 (+ 175us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:12:05.126408 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:12:05.126409 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:12:05.126514 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 21:12:05.126573 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0924 21:12:05.126676 (+ 103us) server_negotiation.cc:299] Negotiation successful
0924 21:12:05.126739 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260924 21:12:05.126816 22635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:12:05.120038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36999 (local address 127.0.0.1:58558)
0924 21:12:05.120080 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0924 21:12:05.120089 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 21:12:05.120122 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 21:12:05.120238 (+ 116us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 21:12:05.120239 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 21:12:05.120384 (+ 145us) client_negotiation.cc:189] Negotiated authn=SASL
0924 21:12:05.120422 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:12:05.120424 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:12:05.121295 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:12:05.121296 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:12:05.122108 (+ 812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:12:05.122111 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:12:05.122510 (+ 399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:12:05.122511 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:12:05.122564 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:12:05.122930 (+ 366us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 21:12:05.122935 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 21:12:05.124172 (+ 1237us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 21:12:05.125876 (+ 1704us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 21:12:05.125876 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 21:12:05.125877 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 21:12:05.126080 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 21:12:05.126283 (+ 203us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 21:12:05.126284 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 21:12:05.126284 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 21:12:05.126340 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 21:12:05.126563 (+ 223us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 21:12:05.126563 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 21:12:05.126600 (+ 37us) client_negotiation.cc:769] Sending connection context
0924 21:12:05.126658 (+ 58us) client_negotiation.cc:240] Negotiation successful
0924 21:12:05.126735 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260924 21:12:05.128455 22633 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 024fa4c9c5594295af50083f4f3e4289: Bootstrap starting.
I20260924 21:12:05.129104 22633 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 024fa4c9c5594295af50083f4f3e4289: Neither blocks nor log segments found. Creating new log.
I20260924 21:12:05.129370 22633 log.cc:824] T 00000000000000000000000000000000 P 024fa4c9c5594295af50083f4f3e4289: Log is configured to *not* fsync() on all Append() calls
I20260924 21:12:05.130136 22633 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 024fa4c9c5594295af50083f4f3e4289: No bootstrap required, opened a new log
I20260924 21:12:05.131419 22633 raft_consensus.cc:348] T 00000000000000000000000000000000 P 024fa4c9c5594295af50083f4f3e4289 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "024fa4c9c5594295af50083f4f3e4289" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36755 } } peers { permanent_uuid: "04b4ac56836d40a7926274191e6e3744" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41801 } } peers { permanent_uuid: "79a0407db57b41b8a3813815f5a8cfc6" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36999 } }
I20260924 21:12:05.131543 22633 raft_consensus.cc:729] T 00000000000000000000000000000000 P 024fa4c9c5594295af50083f4f3e4289 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 024fa4c9c5594295af50083f4f3e4289, State: Initialized, Role: FOLLOWER
I20260924 21:12:05.131616 22633 consensus_queue.cc:260] T 00000000000000000000000000000000 P 024fa4c9c5594295af50083f4f3e4289 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "024fa4c9c5594295af50083f4f3e4289" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36755 } } peers { permanent_uuid: "04b4ac56836d40a7926274191e6e3744" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41801 } } peers { permanent_uuid: "79a0407db57b41b8a3813815f5a8cfc6" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36999 } }
I20260924 21:12:05.131883 22633 sys_catalog.cc:564] T 00000000000000000000000000000000 P 024fa4c9c5594295af50083f4f3e4289 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 21:12:05.131896 22607 raft_consensus.cc:482] T 00000000000000000000000000000000 P 024fa4c9c5594295af50083f4f3e4289 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260924 21:12:05.131980 22607 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 024fa4c9c5594295af50083f4f3e4289 [term 0 FOLLOWER]: Advancing to term 1
I20260924 21:12:05.132715 22607 raft_consensus.cc:504] T 00000000000000000000000000000000 P 024fa4c9c5594295af50083f4f3e4289 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "024fa4c9c5594295af50083f4f3e4289" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36755 } } peers { permanent_uuid: "04b4ac56836d40a7926274191e6e3744" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41801 } } peers { permanent_uuid: "79a0407db57b41b8a3813815f5a8cfc6" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36999 } }
I20260924 21:12:05.132913 22607 leader_election.cc:288] T 00000000000000000000000000000000 P 024fa4c9c5594295af50083f4f3e4289 [CANDIDATE]: Term 1 election: Requested vote from peers 04b4ac56836d40a7926274191e6e3744 (127.0.98.61:41801), 79a0407db57b41b8a3813815f5a8cfc6 (127.0.98.60:36999)
I20260924 21:12:05.133363 22682 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "024fa4c9c5594295af50083f4f3e4289" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "04b4ac56836d40a7926274191e6e3744"
I20260924 21:12:05.133490 22682 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 04b4ac56836d40a7926274191e6e3744 [term 0 FOLLOWER]: Advancing to term 1
I20260924 21:12:05.134295 22682 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 04b4ac56836d40a7926274191e6e3744 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 024fa4c9c5594295af50083f4f3e4289 in term 1.
I20260924 21:12:05.134508 22757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "024fa4c9c5594295af50083f4f3e4289" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "79a0407db57b41b8a3813815f5a8cfc6"
I20260924 21:12:05.134624 22757 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 79a0407db57b41b8a3813815f5a8cfc6 [term 0 FOLLOWER]: Advancing to term 1
I20260924 21:12:05.135357 22757 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 79a0407db57b41b8a3813815f5a8cfc6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 024fa4c9c5594295af50083f4f3e4289 in term 1.
W20260924 21:12:05.135694 22829 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 024fa4c9c5594295af50083f4f3e4289: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 21:12:05.135811 22829 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 21:12:05.136235 22583 leader_election.cc:302] T 00000000000000000000000000000000 P 024fa4c9c5594295af50083f4f3e4289 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 024fa4c9c5594295af50083f4f3e4289, 04b4ac56836d40a7926274191e6e3744; no voters:
I20260924 21:12:05.137701 22565 master_runner.cc:263] No leader master found from master 024fa4c9c5594295af50083f4f3e4289
I20260924 21:12:05.137763 22565 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260924 21:12:05.131907 22818 sys_catalog.cc:455] T 00000000000000000000000000000000 P 024fa4c9c5594295af50083f4f3e4289 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "024fa4c9c5594295af50083f4f3e4289" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36755 } } peers { permanent_uuid: "04b4ac56836d40a7926274191e6e3744" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41801 } } peers { permanent_uuid: "79a0407db57b41b8a3813815f5a8cfc6" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36999 } } }
I20260924 21:12:05.137894 22818 sys_catalog.cc:458] T 00000000000000000000000000000000 P 024fa4c9c5594295af50083f4f3e4289 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 21:12:05.137975 22818 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 024fa4c9c5594295af50083f4f3e4289 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 21:12:05.138021 22818 raft_consensus.cc:686] T 00000000000000000000000000000000 P 024fa4c9c5594295af50083f4f3e4289 [term 1 LEADER]: Becoming Leader. State: Replica: 024fa4c9c5594295af50083f4f3e4289, State: Running, Role: LEADER
I20260924 21:12:05.138094 22818 consensus_queue.cc:237] T 00000000000000000000000000000000 P 024fa4c9c5594295af50083f4f3e4289 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "024fa4c9c5594295af50083f4f3e4289" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36755 } } peers { permanent_uuid: "04b4ac56836d40a7926274191e6e3744" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41801 } } peers { permanent_uuid: "79a0407db57b41b8a3813815f5a8cfc6" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36999 } }
I20260924 21:12:05.138499 22818 sys_catalog.cc:455] T 00000000000000000000000000000000 P 024fa4c9c5594295af50083f4f3e4289 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 024fa4c9c5594295af50083f4f3e4289. Latest consensus state: current_term: 1 leader_uuid: "024fa4c9c5594295af50083f4f3e4289" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "024fa4c9c5594295af50083f4f3e4289" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36755 } } peers { permanent_uuid: "04b4ac56836d40a7926274191e6e3744" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41801 } } peers { permanent_uuid: "79a0407db57b41b8a3813815f5a8cfc6" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36999 } } }
I20260924 21:12:05.138571 22818 sys_catalog.cc:458] T 00000000000000000000000000000000 P 024fa4c9c5594295af50083f4f3e4289 [sys.catalog]: This master's current role is: LEADER
I20260924 21:12:05.139364 22837 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 21:12:05.139576 22837 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 21:12:05.145792 22757 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 79a0407db57b41b8a3813815f5a8cfc6 [term 1 FOLLOWER]: Refusing update from remote peer 024fa4c9c5594295af50083f4f3e4289: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 21:12:05.146148 22835 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 024fa4c9c5594295af50083f4f3e4289 [LEADER]: Connected to new peer: Peer: permanent_uuid: "79a0407db57b41b8a3813815f5a8cfc6" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36999 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 21:12:05.147967 22788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:12:05.139132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58568 (local address 127.0.98.60:36999)
0924 21:12:05.139163 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0924 21:12:05.139165 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 21:12:05.141129 (+ 1964us) server_negotiation.cc:407] Connection header received
0924 21:12:05.141213 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:12:05.141214 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:12:05.141222 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:12:05.141251 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:12:05.141928 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:12:05.142616 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:12:05.143550 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:12:05.143839 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:12:05.143873 (+ 34us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:12:05.145371 (+ 1498us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 21:12:05.145372 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 21:12:05.145372 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 21:12:05.145378 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 21:12:05.146878 (+ 1500us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:12:05.147235 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:12:05.147236 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:12:05.147237 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:12:05.147283 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:12:05.147492 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:12:05.147493 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:12:05.147493 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:12:05.147596 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 21:12:05.147658 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0924 21:12:05.147831 (+ 173us) server_negotiation.cc:299] Negotiation successful
0924 21:12:05.147892 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260924 21:12:05.147995 22804 sys_catalog.cc:455] T 00000000000000000000000000000000 P 79a0407db57b41b8a3813815f5a8cfc6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 024fa4c9c5594295af50083f4f3e4289. Latest consensus state: current_term: 1 leader_uuid: "024fa4c9c5594295af50083f4f3e4289" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "024fa4c9c5594295af50083f4f3e4289" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36755 } } peers { permanent_uuid: "04b4ac56836d40a7926274191e6e3744" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41801 } } peers { permanent_uuid: "79a0407db57b41b8a3813815f5a8cfc6" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36999 } } }
I20260924 21:12:05.148068 22804 sys_catalog.cc:458] T 00000000000000000000000000000000 P 79a0407db57b41b8a3813815f5a8cfc6 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 21:12:05.148108 22682 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 04b4ac56836d40a7926274191e6e3744 [term 1 FOLLOWER]: Refusing update from remote peer 024fa4c9c5594295af50083f4f3e4289: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 21:12:05.148579 22835 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 024fa4c9c5594295af50083f4f3e4289 [LEADER]: Connected to new peer: Peer: permanent_uuid: "04b4ac56836d40a7926274191e6e3744" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41801 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 21:12:05.150393 22790 sys_catalog.cc:455] T 00000000000000000000000000000000 P 04b4ac56836d40a7926274191e6e3744 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 024fa4c9c5594295af50083f4f3e4289. Latest consensus state: current_term: 1 leader_uuid: "024fa4c9c5594295af50083f4f3e4289" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "024fa4c9c5594295af50083f4f3e4289" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36755 } } peers { permanent_uuid: "04b4ac56836d40a7926274191e6e3744" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41801 } } peers { permanent_uuid: "79a0407db57b41b8a3813815f5a8cfc6" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36999 } } }
I20260924 21:12:05.150475 22790 sys_catalog.cc:458] T 00000000000000000000000000000000 P 04b4ac56836d40a7926274191e6e3744 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 21:12:05.152472 22840 mvcc.cc:205] Tried to move back new op lower bound from 7333004595795021824 to 7333004595766358016. Current Snapshot: MvccSnapshot[applied={T|T < 7333004595795021824}]
I20260924 21:12:05.152822 22804 sys_catalog.cc:455] T 00000000000000000000000000000000 P 79a0407db57b41b8a3813815f5a8cfc6 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "024fa4c9c5594295af50083f4f3e4289" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "024fa4c9c5594295af50083f4f3e4289" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36755 } } peers { permanent_uuid: "04b4ac56836d40a7926274191e6e3744" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41801 } } peers { permanent_uuid: "79a0407db57b41b8a3813815f5a8cfc6" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36999 } } }
I20260924 21:12:05.152904 22804 sys_catalog.cc:458] T 00000000000000000000000000000000 P 79a0407db57b41b8a3813815f5a8cfc6 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 21:12:05.153071 22835 sys_catalog.cc:455] T 00000000000000000000000000000000 P 024fa4c9c5594295af50083f4f3e4289 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "024fa4c9c5594295af50083f4f3e4289" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "024fa4c9c5594295af50083f4f3e4289" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36755 } } peers { permanent_uuid: "04b4ac56836d40a7926274191e6e3744" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41801 } } peers { permanent_uuid: "79a0407db57b41b8a3813815f5a8cfc6" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36999 } } }
I20260924 21:12:05.153167 22835 sys_catalog.cc:458] T 00000000000000000000000000000000 P 024fa4c9c5594295af50083f4f3e4289 [sys.catalog]: This master's current role is: LEADER
I20260924 21:12:05.153249 22842 mvcc.cc:205] Tried to move back new op lower bound from 7333004595795021824 to 7333004595766358016. Current Snapshot: MvccSnapshot[applied={T|T < 7333004595795021824}]
I20260924 21:12:05.153388 22713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:12:05.138952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37900 (local address 127.0.98.61:41801)
0924 21:12:05.138988 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0924 21:12:05.138990 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 21:12:05.140983 (+ 1993us) server_negotiation.cc:407] Connection header received
0924 21:12:05.141054 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:12:05.141055 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:12:05.141065 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:12:05.141126 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:12:05.146767 (+ 5641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:12:05.147491 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:12:05.148854 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:12:05.149082 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:12:05.149123 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:12:05.150580 (+ 1457us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 21:12:05.150582 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 21:12:05.150582 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 21:12:05.150588 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 21:12:05.152256 (+ 1668us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:12:05.152606 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:12:05.152607 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:12:05.152608 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:12:05.152681 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:12:05.152876 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:12:05.152876 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:12:05.152877 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:12:05.152981 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 21:12:05.153053 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0924 21:12:05.153238 (+ 185us) server_negotiation.cc:299] Negotiation successful
0924 21:12:05.153308 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 21:12:05.157384 22636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:12:05.140844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59064 (local address 127.0.98.62:36755)
0924 21:12:05.140877 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0924 21:12:05.140879 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 21:12:05.148604 (+ 7725us) server_negotiation.cc:407] Connection header received
0924 21:12:05.148687 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:12:05.148688 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:12:05.148697 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:12:05.148723 (+ 26us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:12:05.149261 (+ 538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:12:05.149915 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:12:05.150749 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:12:05.151049 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:12:05.151087 (+ 38us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:12:05.154920 (+ 3833us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 21:12:05.154922 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 21:12:05.154922 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 21:12:05.154928 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 21:12:05.156374 (+ 1446us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:12:05.156738 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:12:05.156739 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:12:05.156739 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:12:05.156780 (+ 41us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:12:05.156934 (+ 154us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:12:05.156935 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:12:05.156935 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:12:05.157034 (+ 99us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 21:12:05.157089 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0924 21:12:05.157255 (+ 166us) server_negotiation.cc:299] Negotiation successful
0924 21:12:05.157313 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260924 21:12:05.158160 22837 catalog_manager.cc:1347] Generated new cluster ID: dbb1c0cd83e341d8a0fa8baaf85424e7
I20260924 21:12:05.158209 22837 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 21:12:05.162268 22835 sys_catalog.cc:455] T 00000000000000000000000000000000 P 024fa4c9c5594295af50083f4f3e4289 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "024fa4c9c5594295af50083f4f3e4289" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "024fa4c9c5594295af50083f4f3e4289" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36755 } } peers { permanent_uuid: "04b4ac56836d40a7926274191e6e3744" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41801 } } peers { permanent_uuid: "79a0407db57b41b8a3813815f5a8cfc6" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36999 } } }
I20260924 21:12:05.162336 22835 sys_catalog.cc:458] T 00000000000000000000000000000000 P 024fa4c9c5594295af50083f4f3e4289 [sys.catalog]: This master's current role is: LEADER
I20260924 21:12:05.162750 22790 sys_catalog.cc:455] T 00000000000000000000000000000000 P 04b4ac56836d40a7926274191e6e3744 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "024fa4c9c5594295af50083f4f3e4289" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "024fa4c9c5594295af50083f4f3e4289" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36755 } } peers { permanent_uuid: "04b4ac56836d40a7926274191e6e3744" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41801 } } peers { permanent_uuid: "79a0407db57b41b8a3813815f5a8cfc6" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36999 } } }
I20260924 21:12:05.162818 22790 sys_catalog.cc:458] T 00000000000000000000000000000000 P 04b4ac56836d40a7926274191e6e3744 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 21:12:05.162959 22844 mvcc.cc:205] Tried to move back new op lower bound from 7333004595795021824 to 7333004595766358016. Current Snapshot: MvccSnapshot[applied={T|T < 7333004595795021824}]
I20260924 21:12:05.173903 22837 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 21:12:05.174548 22837 catalog_manager.cc:1504] Loading token signing keys...
I20260924 21:12:05.179840 22837 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 024fa4c9c5594295af50083f4f3e4289: Generated new TSK 0
I20260924 21:12:05.179975 22837 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260924 21:12:05.201196 22713 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 21:12:05.200014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37906 (local address 127.0.98.61:41801)
0924 21:12:05.200047 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0924 21:12:05.200049 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 21:12:05.200563 (+ 514us) server_negotiation.cc:407] Connection header received
0924 21:12:05.200658 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:12:05.200659 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:12:05.200668 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:12:05.200696 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:12:05.201138 (+ 442us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37906: BlockingRecv error: recv got EOF from 127.0.0.1:37906 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260924 21:12:05.202170 22788 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 21:12:05.200303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58572 (local address 127.0.98.60:36999)
0924 21:12:05.200335 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0924 21:12:05.200337 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 21:12:05.201414 (+ 1077us) server_negotiation.cc:407] Connection header received
0924 21:12:05.201724 (+ 310us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:12:05.201725 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:12:05.201735 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:12:05.201764 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:12:05.202124 (+ 360us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58572: BlockingRecv error: recv got EOF from 127.0.0.1:58572 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260924 21:12:05.202893 22636 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 21:12:05.201586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59072 (local address 127.0.98.62:36755)
0924 21:12:05.201621 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0924 21:12:05.201623 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 21:12:05.202375 (+ 752us) server_negotiation.cc:407] Connection header received
0924 21:12:05.202528 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:12:05.202529 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:12:05.202539 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:12:05.202565 (+ 26us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:12:05.202846 (+ 281us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59072: BlockingRecv error: recv got EOF from 127.0.0.1:59072 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260924 21:12:05.205034 22713 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 21:12:05.204290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37912 (local address 127.0.98.61:41801)
0924 21:12:05.204323 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0924 21:12:05.204324 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 21:12:05.204534 (+ 210us) server_negotiation.cc:407] Connection header received
0924 21:12:05.204632 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:12:05.204633 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:12:05.204648 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:12:05.204676 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:12:05.204985 (+ 309us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37912: BlockingRecv error: recv got EOF from 127.0.0.1:37912 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260924 21:12:05.207729 22713 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 21:12:05.207044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37926 (local address 127.0.98.61:41801)
0924 21:12:05.207069 (+ 25us) server_negotiation.cc:206] Beginning negotiation
0924 21:12:05.207070 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 21:12:05.207279 (+ 209us) server_negotiation.cc:407] Connection header received
0924 21:12:05.207375 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:12:05.207376 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:12:05.207385 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:12:05.207410 (+ 25us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:12:05.207696 (+ 286us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37926: BlockingRecv error: recv got EOF from 127.0.0.1:37926 (error 108)
Metrics: {"server-negotiator.queue_time_us":12}
W20260924 21:12:05.211309 22713 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 21:12:05.210581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37932 (local address 127.0.98.61:41801)
0924 21:12:05.210609 (+ 28us) server_negotiation.cc:206] Beginning negotiation
0924 21:12:05.210610 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 21:12:05.210853 (+ 243us) server_negotiation.cc:407] Connection header received
0924 21:12:05.210947 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:12:05.210948 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:12:05.210957 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:12:05.210985 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:12:05.211273 (+ 288us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37932: BlockingRecv error: recv got EOF from 127.0.0.1:37932 (error 108)
Metrics: {"server-negotiator.queue_time_us":14}
W20260924 21:12:05.216799 22713 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 21:12:05.216165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37934 (local address 127.0.98.61:41801)
0924 21:12:05.216191 (+ 26us) server_negotiation.cc:206] Beginning negotiation
0924 21:12:05.216193 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 21:12:05.216389 (+ 196us) server_negotiation.cc:407] Connection header received
0924 21:12:05.216482 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:12:05.216482 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:12:05.216490 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:12:05.216513 (+ 23us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:12:05.216762 (+ 249us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37934: BlockingRecv error: recv got EOF from 127.0.0.1:37934 (error 108)
Metrics: {"server-negotiator.queue_time_us":14}
W20260924 21:12:05.226277 22713 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 21:12:05.225629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37950 (local address 127.0.98.61:41801)
0924 21:12:05.225656 (+ 27us) server_negotiation.cc:206] Beginning negotiation
0924 21:12:05.225657 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 21:12:05.225884 (+ 227us) server_negotiation.cc:407] Connection header received
0924 21:12:05.225964 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:12:05.225964 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:12:05.225972 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:12:05.225995 (+ 23us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:12:05.226245 (+ 250us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37950: BlockingRecv error: recv got EOF from 127.0.0.1:37950 (error 108)
Metrics: {"server-negotiator.queue_time_us":13}
W20260924 21:12:05.243863 22713 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 21:12:05.243192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37954 (local address 127.0.98.61:41801)
0924 21:12:05.243221 (+ 29us) server_negotiation.cc:206] Beginning negotiation
0924 21:12:05.243223 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 21:12:05.243438 (+ 215us) server_negotiation.cc:407] Connection header received
0924 21:12:05.243514 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:12:05.243515 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:12:05.243523 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:12:05.243546 (+ 23us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:12:05.243823 (+ 277us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37954: BlockingRecv error: recv got EOF from 127.0.0.1:37954 (error 108)
Metrics: {"server-negotiator.queue_time_us":14}
W20260924 21:12:05.277712 22713 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 21:12:05.277030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37968 (local address 127.0.98.61:41801)
0924 21:12:05.277084 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0924 21:12:05.277087 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 21:12:05.277205 (+ 118us) server_negotiation.cc:407] Connection header received
0924 21:12:05.277279 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:12:05.277280 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:12:05.277291 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:12:05.277357 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:12:05.277639 (+ 282us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37968: BlockingRecv error: recv got EOF from 127.0.0.1:37968 (error 108)
Metrics: {"server-negotiator.queue_time_us":28}
W20260924 21:12:05.343657 22713 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 21:12:05.342883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37974 (local address 127.0.98.61:41801)
0924 21:12:05.342953 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0924 21:12:05.342956 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 21:12:05.343057 (+ 101us) server_negotiation.cc:407] Connection header received
0924 21:12:05.343144 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:12:05.343145 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:12:05.343163 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:12:05.343226 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:12:05.343590 (+ 364us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37974: BlockingRecv error: recv got EOF from 127.0.0.1:37974 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
W20260924 21:12:05.473651 22713 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 21:12:05.472822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37988 (local address 127.0.98.61:41801)
0924 21:12:05.472900 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0924 21:12:05.472902 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 21:12:05.473010 (+ 108us) server_negotiation.cc:407] Connection header received
0924 21:12:05.473120 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:12:05.473122 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:12:05.473140 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:12:05.473210 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:12:05.473576 (+ 366us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37988: BlockingRecv error: recv got EOF from 127.0.0.1:37988 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
W20260924 21:12:05.731551 22713 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 21:12:05.730746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37994 (local address 127.0.98.61:41801)
0924 21:12:05.730818 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0924 21:12:05.730820 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 21:12:05.730829 (+ 9us) server_negotiation.cc:407] Connection header received
0924 21:12:05.730985 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:12:05.730986 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:12:05.731009 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:12:05.731082 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:12:05.731480 (+ 398us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37994: BlockingRecv error: recv got EOF from 127.0.0.1:37994 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
I20260924 21:12:06.082935 22801 catalog_manager.cc:1259] Loaded cluster ID: dbb1c0cd83e341d8a0fa8baaf85424e7
I20260924 21:12:06.083021 22801 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 04b4ac56836d40a7926274191e6e3744: loading cluster ID for follower catalog manager: success
I20260924 21:12:06.083707 22801 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 04b4ac56836d40a7926274191e6e3744: acquiring CA information for follower catalog manager: success
I20260924 21:12:06.084067 22801 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 04b4ac56836d40a7926274191e6e3744: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 21:12:06.094782 22815 catalog_manager.cc:1259] Loaded cluster ID: dbb1c0cd83e341d8a0fa8baaf85424e7
I20260924 21:12:06.094827 22815 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 79a0407db57b41b8a3813815f5a8cfc6: loading cluster ID for follower catalog manager: success
I20260924 21:12:06.095441 22815 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 79a0407db57b41b8a3813815f5a8cfc6: acquiring CA information for follower catalog manager: success
I20260924 21:12:06.095628 22815 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 79a0407db57b41b8a3813815f5a8cfc6: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 21:12:06.247771 22914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:12:06.244806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38010 (local address 127.0.98.61:41801)
0924 21:12:06.244930 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0924 21:12:06.244932 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 21:12:06.245024 (+ 92us) server_negotiation.cc:407] Connection header received
0924 21:12:06.245101 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:12:06.245102 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:12:06.245149 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:12:06.245234 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 21:12:06.245379 (+ 145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:12:06.245936 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:12:06.246846 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:12:06.247170 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:12:06.247247 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:12:06.247463 (+ 216us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 21:12:06.247573 (+ 110us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 21:12:06.247638 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0924 21:12:06.247676 (+ 38us) server_negotiation.cc:299] Negotiation successful
0924 21:12:06.247707 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87,"thread_start_us":51,"threads_started":1}
I20260924 21:12:06.251569 22920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:12:06.249401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58582 (local address 127.0.98.60:36999)
0924 21:12:06.249486 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0924 21:12:06.249487 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 21:12:06.249493 (+ 6us) server_negotiation.cc:407] Connection header received
0924 21:12:06.249526 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:12:06.249527 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:12:06.249555 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:12:06.249616 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 21:12:06.249725 (+ 109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:12:06.250235 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:12:06.250882 (+ 647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:12:06.251079 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:12:06.251125 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:12:06.251250 (+ 125us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 21:12:06.251329 (+ 79us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 21:12:06.251381 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0924 21:12:06.251444 (+ 63us) server_negotiation.cc:299] Negotiation successful
0924 21:12:06.251458 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71,"thread_start_us":41,"threads_started":1}
I20260924 21:12:06.255573 22914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:12:06.253000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38022 (local address 127.0.98.61:41801)
0924 21:12:06.253034 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0924 21:12:06.253035 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 21:12:06.253215 (+ 180us) server_negotiation.cc:407] Connection header received
0924 21:12:06.253241 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:12:06.253242 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:12:06.253252 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:12:06.253300 (+ 48us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 21:12:06.253382 (+ 82us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:12:06.254016 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:12:06.254884 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:12:06.255121 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:12:06.255185 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:12:06.255330 (+ 145us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 21:12:06.255359 (+ 29us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 21:12:06.255430 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0924 21:12:06.255477 (+ 47us) server_negotiation.cc:299] Negotiation successful
0924 21:12:06.255495 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260924 21:12:06.256096 22920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:12:06.253308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58590 (local address 127.0.98.60:36999)
0924 21:12:06.253341 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0924 21:12:06.253343 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 21:12:06.253502 (+ 159us) server_negotiation.cc:407] Connection header received
0924 21:12:06.253599 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:12:06.253600 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:12:06.253610 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:12:06.253668 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 21:12:06.253773 (+ 105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:12:06.254444 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:12:06.255221 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:12:06.255505 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:12:06.255558 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:12:06.255726 (+ 168us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 21:12:06.255754 (+ 28us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 21:12:06.255802 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0924 21:12:06.255999 (+ 197us) server_negotiation.cc:299] Negotiation successful
0924 21:12:06.256043 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260924 21:12:06.256176 22926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:12:06.252972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59088 (local address 127.0.98.62:36755)
0924 21:12:06.253142 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0924 21:12:06.253144 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 21:12:06.253152 (+ 8us) server_negotiation.cc:407] Connection header received
0924 21:12:06.253190 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:12:06.253191 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:12:06.253236 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:12:06.253316 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 21:12:06.253481 (+ 165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:12:06.254285 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:12:06.255233 (+ 948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:12:06.255548 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:12:06.255603 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:12:06.255801 (+ 198us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 21:12:06.255889 (+ 88us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 21:12:06.255957 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0924 21:12:06.256063 (+ 106us) server_negotiation.cc:299] Negotiation successful
0924 21:12:06.256086 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149,"thread_start_us":45,"threads_started":1}
I20260924 21:12:06.257738 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAzNNjS/build/release/bin/kudu with pid 22565
I20260924 21:12:06.262671 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAzNNjS/build/release/bin/kudu with pid 22640
I20260924 21:12:06.267042 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAzNNjS/build/release/bin/kudu with pid 22715
2026-09-24T21:12:06Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (2656 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790284076800039-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 21:12:06 dist-test-slave-c7tk krb5kdc[22931](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 21:12:06 dist-test-slave-c7tk krb5kdc[22931](info): set up 2 sockets
Sep 24 21:12:06 dist-test-slave-c7tk krb5kdc[22931](info): commencing operation
krb5kdc: starting...
W20260924 21:12:07.193735 392 mini_kdc.cc:121] Time spent starting KDC: real 0.908s user 0.000s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790284076800039-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790284076800039-392-0/krb5kdc/test-user.keytab.
Sep 24 21:12:07 dist-test-slave-c7tk krb5kdc[22931](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284327, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T21:12:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T21:12:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790284076800039-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790284076800039-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 21:12:07.294376 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAzNNjS/build/release/bin/kudu
/tmp/dist-test-taskAzNNjS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790284076800039-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790284076800039-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790284076800039-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790284076800039-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45909
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790284076800039-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:32789
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45909,127.0.98.61:38943,127.0.98.60:40187
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790284076800039-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790284076800039-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790284076800039-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 21:12:07.366685 22944 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260924 21:12:07.366853 22944 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 21:12:07.366868 22944 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 21:12:07.366886 22944 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 21:12:07.368003 22944 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 21:12:07.368040 22944 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 21:12:07.368053 22944 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 21:12:07.368098 22944 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 21:12:07.368113 22944 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 21:12:07.369423 22944 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:32789
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790284076800039-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790284076800039-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45909,127.0.98.61:38943,127.0.98.60:40187
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790284076800039-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45909
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790284076800039-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22944
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790284076800039-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 19:49:06 UTC on ccycloud.quasar-vhqjoc.root.comops.site
build id 4670864
I20260924 21:12:07.369642 22944 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 21:12:07.369858 22944 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 21:12:07.374224 22950 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 21:12:07.374228 22952 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 21:12:07.374228 22949 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 21:12:07.374369 22944 server_base.cc:1034] running on GCE node
I20260924 21:12:07.374622 22944 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 21:12:07.374868 22944 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 21:12:07.376003 22944 hybrid_clock.cc:648] HybridClock initialized: now 1790284327376001 us; error 32 us; skew 500 ppm
Sep 24 21:12:07 dist-test-slave-c7tk krb5kdc[22931](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284327, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 21:12:07.378069 22944 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 21:12:07.378437 22944 webserver.cc:460] Webserver started at http://127.0.98.62:38199/ using document root <none> and password file <none>
I20260924 21:12:07.378600 22944 fs_manager.cc:359] Metadata directory not provided
I20260924 21:12:07.378634 22944 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 21:12:07.378705 22944 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 21:12:07.379472 22944 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790284076800039-392-0/minicluster-data/master-0/data/instance:
uuid: "9089a68b94bd4442bc7a5edbfb6ce048"
format_stamp: "Formatted at 2026-09-24 21:12:07 on dist-test-slave-c7tk"
server_key: "a443640d953d8422aa349c1a77f686a0"
server_key_iv: "22c47c1190aa30120c3288d50f6247f4"
server_key_version: "encryptionkey@0"
I20260924 21:12:07.379743 22944 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790284076800039-392-0/minicluster-data/master-0/wal/instance:
uuid: "9089a68b94bd4442bc7a5edbfb6ce048"
format_stamp: "Formatted at 2026-09-24 21:12:07 on dist-test-slave-c7tk"
server_key: "a443640d953d8422aa349c1a77f686a0"
server_key_iv: "22c47c1190aa30120c3288d50f6247f4"
server_key_version: "encryptionkey@0"
I20260924 21:12:07.380837 22944 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20260924 21:12:07.381517 22959 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 21:12:07.381668 22944 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 21:12:07.381752 22944 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790284076800039-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790284076800039-392-0/minicluster-data/master-0/wal
uuid: "9089a68b94bd4442bc7a5edbfb6ce048"
format_stamp: "Formatted at 2026-09-24 21:12:07 on dist-test-slave-c7tk"
server_key: "a443640d953d8422aa349c1a77f686a0"
server_key_iv: "22c47c1190aa30120c3288d50f6247f4"
server_key_version: "encryptionkey@0"
I20260924 21:12:07.381839 22944 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790284076800039-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790284076800039-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790284076800039-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 21:12:07.397218 22944 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 21:12:07.399115 22944 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 21:12:07.399202 22944 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 21:12:07.402566 22944 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45909
I20260924 21:12:07.402628 23011 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45909 every 8 connection(s)
I20260924 21:12:07.402879 22944 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790284076800039-392-0/minicluster-data/master-0/data/info.pb
I20260924 21:12:07.403682 23012 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 21:12:07.405009 23012 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45909 } has no permanent_uuid. Determining permanent_uuid...
I20260924 21:12:07.408051 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAzNNjS/build/release/bin/kudu as pid 22944
I20260924 21:12:07.408121 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790284076800039-392-0/minicluster-data/master-0/wal/instance
I20260924 21:12:07.408315 392 external_mini_cluster.cc:1428] Setting key 8e694e27bf17ae08801eb6305ddcac8a
Sep 24 21:12:07 dist-test-slave-c7tk krb5kdc[22931](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284327, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 21:12:07.413831 23015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:12:07.405675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35936 (local address 127.0.98.62:45909)
0924 21:12:07.405845 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0924 21:12:07.405858 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 21:12:07.405932 (+ 74us) server_negotiation.cc:407] Connection header received
0924 21:12:07.406341 (+ 409us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:12:07.406342 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:12:07.406357 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:12:07.406436 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:12:07.406865 (+ 429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:12:07.407573 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:12:07.408432 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:12:07.408747 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:12:07.408849 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:12:07.410895 (+ 2046us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 21:12:07.410901 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 21:12:07.410901 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 21:12:07.410907 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 21:12:07.412388 (+ 1481us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:12:07.412766 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:12:07.412769 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:12:07.412769 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:12:07.412845 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:12:07.413051 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:12:07.413052 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:12:07.413052 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:12:07.413227 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 21:12:07.413303 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0924 21:12:07.413469 (+ 166us) server_negotiation.cc:299] Negotiation successful
0924 21:12:07.413618 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":20,"server-negotiator.queue_time_us":138,"thread_start_us":93,"threads_started":1}
I20260924 21:12:07.413836 23014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:12:07.405621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45909 (local address 127.0.0.1:35936)
0924 21:12:07.405768 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0924 21:12:07.405844 (+ 76us) client_negotiation.cc:174] Beginning negotiation
0924 21:12:07.406234 (+ 390us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 21:12:07.406459 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 21:12:07.406461 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 21:12:07.406664 (+ 203us) client_negotiation.cc:189] Negotiated authn=SASL
0924 21:12:07.406790 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:12:07.406793 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:12:07.407646 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:12:07.407647 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:12:07.408328 (+ 681us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:12:07.408329 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:12:07.408848 (+ 519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:12:07.408849 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:12:07.408912 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:12:07.409383 (+ 471us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 21:12:07.409392 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 21:12:07.410801 (+ 1409us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 21:12:07.412475 (+ 1674us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 21:12:07.412480 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 21:12:07.412482 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 21:12:07.412679 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 21:12:07.412906 (+ 227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 21:12:07.412906 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 21:12:07.412907 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 21:12:07.412989 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 21:12:07.413295 (+ 306us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 21:12:07.413299 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 21:12:07.413353 (+ 54us) client_negotiation.cc:769] Sending connection context
0924 21:12:07.413448 (+ 95us) client_negotiation.cc:240] Negotiation successful
0924 21:12:07.413660 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"mutex_wait_us":88,"thread_start_us":84,"threads_started":1}
I20260924 21:12:07.414400 23012 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38943 } has no permanent_uuid. Determining permanent_uuid...
I20260924 21:12:07.414734 23014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:12:07.414623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:38943 (local address 127.0.0.1:33714)
0924 21:12:07.414670 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0924 21:12:07.414707 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:38943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260924 21:12:07.414842 22962 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:38943: connect: Connection refused (error 111)
I20260924 21:12:07.415242 23014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:12:07.415159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:38943 (local address 127.0.0.1:33722)
0924 21:12:07.415202 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0924 21:12:07.415223 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:38943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
W20260924 21:12:07.415337 23012 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:38943: Network error: Client connection negotiation failed: client connection to 127.0.98.61:38943: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790284076800039-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790284076800039-392-0/krb5kdc/kudu_127.0.98.61.keytab.
I20260924 21:12:07.430025 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAzNNjS/build/release/bin/kudu
/tmp/dist-test-taskAzNNjS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790284076800039-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790284076800039-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790284076800039-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790284076800039-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:38943
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790284076800039-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:32789
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45909,127.0.98.61:38943,127.0.98.60:40187
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790284076800039-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790284076800039-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790284076800039-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 21:12:07.464635 23012 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38943 } attempt: 1
I20260924 21:12:07.465000 23014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:12:07.464884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:38943 (local address 127.0.0.1:33734)
0924 21:12:07.464936 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0924 21:12:07.464969 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:38943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260924 21:12:07.465387 23014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:12:07.465313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:38943 (local address 127.0.0.1:33738)
0924 21:12:07.465352 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0924 21:12:07.465373 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:38943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260924 21:12:07.465479 23012 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:38943: Network error: Client connection negotiation failed: client connection to 127.0.98.61:38943: connect: Connection refused (error 111)
W20260924 21:12:07.501909 23019 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260924 21:12:07.502086 23019 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 21:12:07.502104 23019 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 21:12:07.502123 23019 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 21:12:07.503083 23019 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 21:12:07.503149 23019 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 21:12:07.503175 23019 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 21:12:07.503192 23019 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 21:12:07.503216 23019 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 21:12:07.504480 23019 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:32789
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790284076800039-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790284076800039-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45909,127.0.98.61:38943,127.0.98.60:40187
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790284076800039-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:38943
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790284076800039-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23019
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790284076800039-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 19:49:06 UTC on ccycloud.quasar-vhqjoc.root.comops.site
build id 4670864
I20260924 21:12:07.504635 23019 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 21:12:07.504771 23019 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 21:12:07.508891 23019 server_base.cc:1034] running on GCE node
W20260924 21:12:07.508858 23025 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 21:12:07.508848 23027 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 21:12:07.508857 23024 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 21:12:07.509245 23019 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 21:12:07.509439 23019 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 21:12:07.510584 23019 hybrid_clock.cc:648] HybridClock initialized: now 1790284327510567 us; error 31 us; skew 500 ppm
Sep 24 21:12:07 dist-test-slave-c7tk krb5kdc[22931](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284327, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 21:12:07.512957 23019 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260924 21:12:07.513336 23019 webserver.cc:460] Webserver started at http://127.0.98.61:41033/ using document root <none> and password file <none>
I20260924 21:12:07.513540 23019 fs_manager.cc:359] Metadata directory not provided
I20260924 21:12:07.513602 23019 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 21:12:07.513737 23019 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 21:12:07.514628 23019 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790284076800039-392-0/minicluster-data/master-1/data/instance:
uuid: "9b75056b27df4d6c88b30ec8b99a0656"
format_stamp: "Formatted at 2026-09-24 21:12:07 on dist-test-slave-c7tk"
server_key: "b3c1416c112a19e5f1df50212eaebe36"
server_key_iv: "3aad3631eee1bcc63a3bf389c439467c"
server_key_version: "encryptionkey@0"
I20260924 21:12:07.514940 23019 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790284076800039-392-0/minicluster-data/master-1/wal/instance:
uuid: "9b75056b27df4d6c88b30ec8b99a0656"
format_stamp: "Formatted at 2026-09-24 21:12:07 on dist-test-slave-c7tk"
server_key: "b3c1416c112a19e5f1df50212eaebe36"
server_key_iv: "3aad3631eee1bcc63a3bf389c439467c"
server_key_version: "encryptionkey@0"
I20260924 21:12:07.516073 23019 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 21:12:07.516676 23034 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 21:12:07.516816 23019 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 21:12:07.516904 23019 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790284076800039-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790284076800039-392-0/minicluster-data/master-1/wal
uuid: "9b75056b27df4d6c88b30ec8b99a0656"
format_stamp: "Formatted at 2026-09-24 21:12:07 on dist-test-slave-c7tk"
server_key: "b3c1416c112a19e5f1df50212eaebe36"
server_key_iv: "3aad3631eee1bcc63a3bf389c439467c"
server_key_version: "encryptionkey@0"
I20260924 21:12:07.516976 23019 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790284076800039-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790284076800039-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790284076800039-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 21:12:07.528996 23019 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 21:12:07.530680 23019 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 21:12:07.530824 23019 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 21:12:07.533630 23012 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38943 } attempt: 2
I20260924 21:12:07.534051 23014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:12:07.533923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:38943 (local address 127.0.0.1:33746)
0924 21:12:07.533984 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0924 21:12:07.534021 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:38943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260924 21:12:07.534394 23014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:12:07.534278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:38943 (local address 127.0.0.1:33762)
0924 21:12:07.534342 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0924 21:12:07.534370 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:38943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260924 21:12:07.534485 23012 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:38943: Network error: Client connection negotiation failed: client connection to 127.0.98.61:38943: connect: Connection refused (error 111)
I20260924 21:12:07.534565 23019 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:38943
I20260924 21:12:07.534597 23086 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:38943 every 8 connection(s)
I20260924 21:12:07.534876 23019 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790284076800039-392-0/minicluster-data/master-1/data/info.pb
I20260924 21:12:07.535738 23087 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 21:12:07.536933 23087 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45909 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 21:12:07 dist-test-slave-c7tk krb5kdc[22931](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284327, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 21:12:07.543702 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAzNNjS/build/release/bin/kudu as pid 23019
I20260924 21:12:07.543795 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790284076800039-392-0/minicluster-data/master-1/wal/instance
I20260924 21:12:07.543984 392 external_mini_cluster.cc:1428] Setting key 99eb6b463b0033cfdbf57a0b0484941c
I20260924 21:12:07.544099 23015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:12:07.537484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35952 (local address 127.0.98.62:45909)
0924 21:12:07.537536 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0924 21:12:07.537538 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 21:12:07.537745 (+ 207us) server_negotiation.cc:407] Connection header received
0924 21:12:07.538133 (+ 388us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:12:07.538134 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:12:07.538144 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:12:07.538187 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:12:07.538645 (+ 458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:12:07.539055 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:12:07.539843 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:12:07.540004 (+ 161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:12:07.540047 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:12:07.541907 (+ 1860us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 21:12:07.541908 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 21:12:07.541908 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 21:12:07.541913 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 21:12:07.542967 (+ 1054us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:12:07.543285 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:12:07.543286 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:12:07.543286 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:12:07.543329 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:12:07.543513 (+ 184us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:12:07.543514 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:12:07.543514 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:12:07.543596 (+ 82us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 21:12:07.543640 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0924 21:12:07.543945 (+ 305us) server_negotiation.cc:299] Negotiation successful
0924 21:12:07.544001 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260924 21:12:07.544269 23089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:12:07.537478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45909 (local address 127.0.0.1:35952)
0924 21:12:07.537636 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0924 21:12:07.537663 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0924 21:12:07.538065 (+ 402us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 21:12:07.538226 (+ 161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 21:12:07.538228 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 21:12:07.538433 (+ 205us) client_negotiation.cc:189] Negotiated authn=SASL
0924 21:12:07.538585 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:12:07.538588 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:12:07.539122 (+ 534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:12:07.539122 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:12:07.539781 (+ 659us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:12:07.539782 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:12:07.540074 (+ 292us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:12:07.540075 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:12:07.540159 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:12:07.540569 (+ 410us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 21:12:07.540576 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0924 21:12:07.541782 (+ 1206us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 21:12:07.543028 (+ 1246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 21:12:07.543034 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 21:12:07.543035 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 21:12:07.543227 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 21:12:07.543379 (+ 152us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 21:12:07.543380 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 21:12:07.543380 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 21:12:07.543459 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 21:12:07.543655 (+ 196us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 21:12:07.543661 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 21:12:07.543803 (+ 142us) client_negotiation.cc:769] Sending connection context
0924 21:12:07.543931 (+ 128us) client_negotiation.cc:240] Negotiation successful
0924 21:12:07.544068 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":90,"threads_started":1}
I20260924 21:12:07.544768 23087 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38943 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 21:12:07 dist-test-slave-c7tk krb5kdc[22931](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284327, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
I20260924 21:12:07.551851 23089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:12:07.544975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:38943 (local address 127.0.0.1:33764)
0924 21:12:07.545008 (+ 33us) negotiation.cc:107] Waiting for socket to connect
0924 21:12:07.545017 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 21:12:07.545044 (+ 27us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 21:12:07.545658 (+ 614us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 21:12:07.545659 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 21:12:07.545820 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0924 21:12:07.545871 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:12:07.545873 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:12:07.546621 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:12:07.546621 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:12:07.547065 (+ 444us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:12:07.547066 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:12:07.547571 (+ 505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:12:07.547571 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:12:07.547627 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:12:07.547984 (+ 357us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 21:12:07.547987 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0924 21:12:07.549161 (+ 1174us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 21:12:07.550781 (+ 1620us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 21:12:07.550781 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 21:12:07.550782 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 21:12:07.550930 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 21:12:07.551208 (+ 278us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 21:12:07.551209 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 21:12:07.551210 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 21:12:07.551262 (+ 52us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 21:12:07.551527 (+ 265us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 21:12:07.551528 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 21:12:07.551563 (+ 35us) client_negotiation.cc:769] Sending connection context
0924 21:12:07.551633 (+ 70us) client_negotiation.cc:240] Negotiation successful
0924 21:12:07.551720 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":21}
I20260924 21:12:07.551872 23091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:12:07.545407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33764 (local address 127.0.98.61:38943)
0924 21:12:07.545535 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0924 21:12:07.545539 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 21:12:07.545551 (+ 12us) server_negotiation.cc:407] Connection header received
0924 21:12:07.545566 (+ 15us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:12:07.545567 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:12:07.545575 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:12:07.545628 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:12:07.545937 (+ 309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:12:07.546518 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:12:07.547150 (+ 632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:12:07.547495 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:12:07.547570 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:12:07.549256 (+ 1686us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 21:12:07.549262 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 21:12:07.549263 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 21:12:07.549268 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 21:12:07.550694 (+ 1426us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:12:07.551048 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:12:07.551050 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:12:07.551050 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:12:07.551138 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:12:07.551341 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:12:07.551342 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:12:07.551342 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:12:07.551460 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 21:12:07.551545 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0924 21:12:07.551659 (+ 114us) server_negotiation.cc:299] Negotiation successful
0924 21:12:07.551784 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":50,"server-negotiator.queue_time_us":99,"thread_start_us":70,"threads_started":1}
I20260924 21:12:07.552418 23087 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40187 } has no permanent_uuid. Determining permanent_uuid...
I20260924 21:12:07.552776 23089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:12:07.552598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40187 (local address 127.0.0.1:56384)
0924 21:12:07.552647 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0924 21:12:07.552760 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40187: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260924 21:12:07.552897 23038 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40187: connect: Connection refused (error 111)
I20260924 21:12:07.553290 23089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:12:07.553156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40187 (local address 127.0.0.1:56392)
0924 21:12:07.553203 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0924 21:12:07.553225 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40187: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
W20260924 21:12:07.553390 23087 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:40187: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40187: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790284076800039-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790284076800039-392-0/krb5kdc/kudu_127.0.98.60.keytab.
I20260924 21:12:07.568549 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAzNNjS/build/release/bin/kudu
/tmp/dist-test-taskAzNNjS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790284076800039-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790284076800039-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790284076800039-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790284076800039-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:40187
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790284076800039-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:32789
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45909,127.0.98.61:38943,127.0.98.60:40187
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790284076800039-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790284076800039-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790284076800039-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 21:12:07.602572 23087 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40187 } attempt: 1
I20260924 21:12:07.603015 23089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:12:07.602894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40187 (local address 127.0.0.1:56404)
0924 21:12:07.602953 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0924 21:12:07.602992 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40187: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260924 21:12:07.603452 23089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:12:07.603260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40187 (local address 127.0.0.1:56420)
0924 21:12:07.603388 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0924 21:12:07.603428 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40187: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112}
W20260924 21:12:07.603583 23087 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:40187: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40187: connect: Connection refused (error 111)
I20260924 21:12:07.625630 23012 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38943 } attempt: 3
Sep 24 21:12:07 dist-test-slave-c7tk krb5kdc[22931](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284327, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260924 21:12:07.632833 23014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:12:07.625956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:38943 (local address 127.0.0.1:33776)
0924 21:12:07.626015 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0924 21:12:07.626026 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 21:12:07.626074 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 21:12:07.626243 (+ 169us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 21:12:07.626243 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 21:12:07.626372 (+ 129us) client_negotiation.cc:189] Negotiated authn=SASL
0924 21:12:07.626438 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:12:07.626439 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:12:07.627422 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:12:07.627423 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:12:07.627914 (+ 491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:12:07.627915 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:12:07.628359 (+ 444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:12:07.628360 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:12:07.628420 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:12:07.628853 (+ 433us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 21:12:07.628857 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 21:12:07.630218 (+ 1361us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 21:12:07.631936 (+ 1718us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 21:12:07.631937 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 21:12:07.631937 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 21:12:07.632085 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 21:12:07.632326 (+ 241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 21:12:07.632326 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 21:12:07.632327 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 21:12:07.632370 (+ 43us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 21:12:07.632638 (+ 268us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 21:12:07.632639 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 21:12:07.632669 (+ 30us) client_negotiation.cc:769] Sending connection context
0924 21:12:07.632714 (+ 45us) client_negotiation.cc:240] Negotiation successful
0924 21:12:07.632775 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39}
I20260924 21:12:07.632892 23091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:12:07.626037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33776 (local address 127.0.98.61:38943)
0924 21:12:07.626096 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0924 21:12:07.626098 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 21:12:07.626109 (+ 11us) server_negotiation.cc:407] Connection header received
0924 21:12:07.626140 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:12:07.626140 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:12:07.626155 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:12:07.626233 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:12:07.626511 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:12:07.627306 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:12:07.627995 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:12:07.628288 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:12:07.628358 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:12:07.630299 (+ 1941us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 21:12:07.630301 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 21:12:07.630301 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 21:12:07.630308 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 21:12:07.631860 (+ 1552us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:12:07.632158 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:12:07.632160 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:12:07.632160 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:12:07.632242 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:12:07.632431 (+ 189us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:12:07.632432 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:12:07.632432 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:12:07.632566 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 21:12:07.632649 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0924 21:12:07.632739 (+ 90us) server_negotiation.cc:299] Negotiation successful
0924 21:12:07.632809 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260924 21:12:07.633333 23012 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40187 } has no permanent_uuid. Determining permanent_uuid...
I20260924 21:12:07.633620 23014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:12:07.633544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40187 (local address 127.0.0.1:56426)
0924 21:12:07.633575 (+ 31us) negotiation.cc:107] Waiting for socket to connect
0924 21:12:07.633601 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40187: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":18}
I20260924 21:12:07.633987 23014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:12:07.633917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40187 (local address 127.0.0.1:56440)
0924 21:12:07.633953 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0924 21:12:07.633974 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40187: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260924 21:12:07.634091 23012 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:40187: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40187: connect: Connection refused (error 111)
W20260924 21:12:07.644133 23094 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260924 21:12:07.644289 23094 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 21:12:07.644305 23094 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 21:12:07.644322 23094 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 21:12:07.645278 23094 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 21:12:07.645309 23094 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 21:12:07.645340 23094 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 21:12:07.645352 23094 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 21:12:07.645363 23094 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 21:12:07.646721 23094 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:32789
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790284076800039-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790284076800039-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45909,127.0.98.61:38943,127.0.98.60:40187
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790284076800039-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:40187
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790284076800039-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23094
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790284076800039-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 19:49:06 UTC on ccycloud.quasar-vhqjoc.root.comops.site
build id 4670864
I20260924 21:12:07.646883 23094 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 21:12:07.647022 23094 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 21:12:07.651288 23094 server_base.cc:1034] running on GCE node
W20260924 21:12:07.651227 23099 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 21:12:07.651224 23100 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 21:12:07.651342 23102 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 21:12:07.651649 23094 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 21:12:07.651935 23094 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 21:12:07.653106 23094 hybrid_clock.cc:648] HybridClock initialized: now 1790284327653103 us; error 42 us; skew 500 ppm
Sep 24 21:12:07 dist-test-slave-c7tk krb5kdc[22931](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284327, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 21:12:07.655162 23094 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260924 21:12:07.655572 23094 webserver.cc:460] Webserver started at http://127.0.98.60:43663/ using document root <none> and password file <none>
I20260924 21:12:07.655779 23094 fs_manager.cc:359] Metadata directory not provided
I20260924 21:12:07.655822 23094 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 21:12:07.655966 23094 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 21:12:07.656903 23094 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790284076800039-392-0/minicluster-data/master-2/data/instance:
uuid: "d1a72b00159e4e9ca4a3dedc22af07e4"
format_stamp: "Formatted at 2026-09-24 21:12:07 on dist-test-slave-c7tk"
server_key: "d5f9319a85baa424179cd78abe49002f"
server_key_iv: "0218931902be28a88225456e1f41cfe2"
server_key_version: "encryptionkey@0"
I20260924 21:12:07.657196 23094 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790284076800039-392-0/minicluster-data/master-2/wal/instance:
uuid: "d1a72b00159e4e9ca4a3dedc22af07e4"
format_stamp: "Formatted at 2026-09-24 21:12:07 on dist-test-slave-c7tk"
server_key: "d5f9319a85baa424179cd78abe49002f"
server_key_iv: "0218931902be28a88225456e1f41cfe2"
server_key_version: "encryptionkey@0"
I20260924 21:12:07.658398 23094 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 21:12:07.659013 23109 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 21:12:07.659178 23094 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 21:12:07.659266 23094 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790284076800039-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790284076800039-392-0/minicluster-data/master-2/wal
uuid: "d1a72b00159e4e9ca4a3dedc22af07e4"
format_stamp: "Formatted at 2026-09-24 21:12:07 on dist-test-slave-c7tk"
server_key: "d5f9319a85baa424179cd78abe49002f"
server_key_iv: "0218931902be28a88225456e1f41cfe2"
server_key_version: "encryptionkey@0"
I20260924 21:12:07.659327 23094 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790284076800039-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790284076800039-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790284076800039-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 21:12:07.665233 23012 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40187 } attempt: 1
I20260924 21:12:07.665542 23014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:12:07.665436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40187 (local address 127.0.0.1:56456)
0924 21:12:07.665485 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0924 21:12:07.665516 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40187: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260924 21:12:07.665898 23014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:12:07.665787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40187 (local address 127.0.0.1:56466)
0924 21:12:07.665857 (+ 70us) negotiation.cc:107] Waiting for socket to connect
0924 21:12:07.665882 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40187: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260924 21:12:07.665982 23012 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:40187: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40187: connect: Connection refused (error 111)
I20260924 21:12:07.667558 23094 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 21:12:07.669049 23094 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 21:12:07.669186 23094 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 21:12:07.671747 23087 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40187 } attempt: 2
I20260924 21:12:07.672111 23089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:12:07.671972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40187 (local address 127.0.0.1:56478)
0924 21:12:07.672032 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0924 21:12:07.672074 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40187: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260924 21:12:07.672479 23089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:12:07.672356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40187 (local address 127.0.0.1:56484)
0924 21:12:07.672416 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0924 21:12:07.672448 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40187: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260924 21:12:07.672586 23087 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:40187: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40187: connect: Connection refused (error 111)
I20260924 21:12:07.673286 23094 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:40187
I20260924 21:12:07.673312 23161 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:40187 every 8 connection(s)
I20260924 21:12:07.673607 23094 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790284076800039-392-0/minicluster-data/master-2/data/info.pb
I20260924 21:12:07.674455 23162 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 21:12:07.675885 23162 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45909 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 21:12:07 dist-test-slave-c7tk krb5kdc[22931](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284327, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 21:12:07.682474 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAzNNjS/build/release/bin/kudu as pid 23094
I20260924 21:12:07.682587 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAzNNjS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790284076800039-392-0/minicluster-data/master-2/wal/instance
I20260924 21:12:07.682780 392 external_mini_cluster.cc:1428] Setting key ffd31bb0af908e0e3db6fda094632a05
I20260924 21:12:07.684734 23015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:12:07.676631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35966 (local address 127.0.98.62:45909)
0924 21:12:07.676690 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0924 21:12:07.676710 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0924 21:12:07.676901 (+ 191us) server_negotiation.cc:407] Connection header received
0924 21:12:07.677420 (+ 519us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:12:07.677421 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:12:07.677434 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:12:07.677484 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:12:07.678147 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:12:07.678601 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:12:07.679558 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:12:07.679738 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:12:07.679826 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:12:07.681829 (+ 2003us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 21:12:07.681830 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 21:12:07.681831 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 21:12:07.681845 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0924 21:12:07.683409 (+ 1564us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:12:07.683794 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:12:07.683795 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:12:07.683796 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:12:07.683880 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:12:07.684155 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:12:07.684156 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:12:07.684156 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:12:07.684291 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 21:12:07.684352 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0924 21:12:07.684623 (+ 271us) server_negotiation.cc:299] Negotiation successful
0924 21:12:07.684682 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260924 21:12:07.684891 23164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:12:07.676603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45909 (local address 127.0.0.1:35966)
0924 21:12:07.676782 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0924 21:12:07.676819 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0924 21:12:07.677307 (+ 488us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 21:12:07.677558 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 21:12:07.677561 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 21:12:07.677769 (+ 208us) client_negotiation.cc:189] Negotiated authn=SASL
0924 21:12:07.678038 (+ 269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:12:07.678043 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:12:07.678711 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:12:07.678714 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:12:07.679453 (+ 739us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:12:07.679455 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:12:07.679859 (+ 404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:12:07.679860 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:12:07.679982 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:12:07.680327 (+ 345us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 21:12:07.680334 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0924 21:12:07.681692 (+ 1358us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 21:12:07.683483 (+ 1791us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 21:12:07.683485 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 21:12:07.683487 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 21:12:07.683710 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 21:12:07.683988 (+ 278us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 21:12:07.683989 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 21:12:07.683989 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 21:12:07.684098 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 21:12:07.684358 (+ 260us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 21:12:07.684362 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 21:12:07.684487 (+ 125us) client_negotiation.cc:769] Sending connection context
0924 21:12:07.684587 (+ 100us) client_negotiation.cc:240] Negotiation successful
0924 21:12:07.684712 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":81,"threads_started":1}
I20260924 21:12:07.685415 23162 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38943 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 21:12:07 dist-test-slave-c7tk krb5kdc[22931](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284327, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Sep 24 21:12:07 dist-test-slave-c7tk krb5kdc[22931](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284327, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260924 21:12:07.691426 23165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:12:07.683145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56498 (local address 127.0.98.60:40187)
0924 21:12:07.683261 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0924 21:12:07.683264 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 21:12:07.683379 (+ 115us) server_negotiation.cc:407] Connection header received
0924 21:12:07.683469 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:12:07.683472 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:12:07.683485 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:12:07.683557 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:12:07.684242 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:12:07.685347 (+ 1105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:12:07.686252 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:12:07.686493 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:12:07.686535 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:12:07.688747 (+ 2212us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 21:12:07.688752 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 21:12:07.688752 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 21:12:07.688757 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 21:12:07.690275 (+ 1518us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:12:07.690608 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:12:07.690610 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:12:07.690610 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:12:07.690664 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:12:07.690850 (+ 186us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:12:07.690850 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:12:07.690851 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:12:07.690991 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 21:12:07.691146 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0924 21:12:07.691232 (+ 86us) server_negotiation.cc:299] Negotiation successful
0924 21:12:07.691312 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93,"thread_start_us":72,"threads_started":1}
I20260924 21:12:07.691625 23091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:12:07.685685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33792 (local address 127.0.98.61:38943)
0924 21:12:07.685730 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0924 21:12:07.685732 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 21:12:07.685761 (+ 29us) server_negotiation.cc:407] Connection header received
0924 21:12:07.685820 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:12:07.685821 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:12:07.685830 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:12:07.685918 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:12:07.686124 (+ 206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:12:07.686740 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:12:07.687364 (+ 624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:12:07.687650 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:12:07.687720 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:12:07.689335 (+ 1615us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 21:12:07.689337 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 21:12:07.689337 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 21:12:07.689343 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 21:12:07.690514 (+ 1171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:12:07.690852 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:12:07.690853 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:12:07.690854 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:12:07.690895 (+ 41us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:12:07.691094 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:12:07.691095 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:12:07.691095 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:12:07.691230 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 21:12:07.691299 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0924 21:12:07.691470 (+ 171us) server_negotiation.cc:299] Negotiation successful
0924 21:12:07.691543 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260924 21:12:07.691620 23164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:12:07.685623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:38943 (local address 127.0.0.1:33792)
0924 21:12:07.685719 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0924 21:12:07.685727 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0924 21:12:07.685766 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 21:12:07.685923 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 21:12:07.685923 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 21:12:07.686020 (+ 97us) client_negotiation.cc:189] Negotiated authn=SASL
0924 21:12:07.686057 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:12:07.686057 (+ 0us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:12:07.686811 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:12:07.686812 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:12:07.687287 (+ 475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:12:07.687288 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:12:07.687729 (+ 441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:12:07.687729 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:12:07.687770 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:12:07.688070 (+ 300us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 21:12:07.688074 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 21:12:07.689252 (+ 1178us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 21:12:07.690584 (+ 1332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 21:12:07.690585 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 21:12:07.690586 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 21:12:07.690788 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 21:12:07.690955 (+ 167us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 21:12:07.690955 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 21:12:07.690956 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 21:12:07.690999 (+ 43us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 21:12:07.691310 (+ 311us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 21:12:07.691311 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 21:12:07.691355 (+ 44us) client_negotiation.cc:769] Sending connection context
0924 21:12:07.691438 (+ 83us) client_negotiation.cc:240] Negotiation successful
0924 21:12:07.691528 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":84}
I20260924 21:12:07.691977 23162 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40187 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 21:12:07 dist-test-slave-c7tk krb5kdc[22931](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284327, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Sep 24 21:12:07 dist-test-slave-c7tk krb5kdc[22931](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284327, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 21:12:07.697942 23164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:12:07.692235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40187 (local address 127.0.0.1:56512)
0924 21:12:07.692283 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0924 21:12:07.692291 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0924 21:12:07.692317 (+ 26us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 21:12:07.692443 (+ 126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 21:12:07.692444 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 21:12:07.692538 (+ 94us) client_negotiation.cc:189] Negotiated authn=SASL
0924 21:12:07.692561 (+ 23us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:12:07.692562 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:12:07.693203 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:12:07.693204 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:12:07.693699 (+ 495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:12:07.693701 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:12:07.694112 (+ 411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:12:07.694113 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:12:07.694157 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:12:07.694642 (+ 485us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 21:12:07.694646 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 21:12:07.695646 (+ 1000us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 21:12:07.697068 (+ 1422us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 21:12:07.697069 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 21:12:07.697069 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 21:12:07.697275 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 21:12:07.697445 (+ 170us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 21:12:07.697446 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 21:12:07.697446 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 21:12:07.697497 (+ 51us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 21:12:07.697672 (+ 175us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 21:12:07.697673 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 21:12:07.697712 (+ 39us) client_negotiation.cc:769] Sending connection context
0924 21:12:07.697782 (+ 70us) client_negotiation.cc:240] Negotiation successful
0924 21:12:07.697896 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38}
I20260924 21:12:07.698031 23165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:12:07.692290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56512 (local address 127.0.98.60:40187)
0924 21:12:07.692318 (+ 28us) server_negotiation.cc:206] Beginning negotiation
0924 21:12:07.692319 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 21:12:07.692326 (+ 7us) server_negotiation.cc:407] Connection header received
0924 21:12:07.692370 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:12:07.692370 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:12:07.692378 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:12:07.692442 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:12:07.692619 (+ 177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:12:07.693111 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:12:07.693772 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:12:07.694050 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:12:07.694091 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:12:07.695729 (+ 1638us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 21:12:07.695730 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 21:12:07.695730 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 21:12:07.695735 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 21:12:07.696987 (+ 1252us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:12:07.697341 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:12:07.697341 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:12:07.697342 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:12:07.697381 (+ 39us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:12:07.697550 (+ 169us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:12:07.697551 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:12:07.697551 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:12:07.697619 (+ 68us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 21:12:07.697656 (+ 37us) server_negotiation.cc:1035] Waiting for connection context
0924 21:12:07.697829 (+ 173us) server_negotiation.cc:299] Negotiation successful
0924 21:12:07.697946 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":53,"server-negotiator.queue_time_us":15}
I20260924 21:12:07.698954 23015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:12:07.692107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35968 (local address 127.0.98.62:45909)
0924 21:12:07.692143 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0924 21:12:07.692145 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 21:12:07.692153 (+ 8us) server_negotiation.cc:407] Connection header received
0924 21:12:07.692175 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:12:07.692175 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:12:07.692185 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:12:07.692245 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:12:07.692800 (+ 555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:12:07.693249 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:12:07.693891 (+ 642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:12:07.694064 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:12:07.694143 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:12:07.696255 (+ 2112us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 21:12:07.696256 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 21:12:07.696256 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 21:12:07.696262 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 21:12:07.697768 (+ 1506us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:12:07.698135 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:12:07.698137 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:12:07.698137 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:12:07.698208 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:12:07.698452 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:12:07.698452 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:12:07.698453 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:12:07.698617 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 21:12:07.698767 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0924 21:12:07.698801 (+ 34us) server_negotiation.cc:299] Negotiation successful
0924 21:12:07.698864 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260924 21:12:07.699308 23162 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d1a72b00159e4e9ca4a3dedc22af07e4: Bootstrap starting.
I20260924 21:12:07.700268 23162 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d1a72b00159e4e9ca4a3dedc22af07e4: Neither blocks nor log segments found. Creating new log.
I20260924 21:12:07.700541 23162 log.cc:824] T 00000000000000000000000000000000 P d1a72b00159e4e9ca4a3dedc22af07e4: Log is configured to *not* fsync() on all Append() calls
I20260924 21:12:07.701290 23162 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d1a72b00159e4e9ca4a3dedc22af07e4: No bootstrap required, opened a new log
I20260924 21:12:07.703382 23162 raft_consensus.cc:348] T 00000000000000000000000000000000 P d1a72b00159e4e9ca4a3dedc22af07e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9089a68b94bd4442bc7a5edbfb6ce048" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45909 } } peers { permanent_uuid: "9b75056b27df4d6c88b30ec8b99a0656" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38943 } } peers { permanent_uuid: "d1a72b00159e4e9ca4a3dedc22af07e4" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40187 } }
I20260924 21:12:07.703477 23162 raft_consensus.cc:729] T 00000000000000000000000000000000 P d1a72b00159e4e9ca4a3dedc22af07e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1a72b00159e4e9ca4a3dedc22af07e4, State: Initialized, Role: FOLLOWER
I20260924 21:12:07.703557 23162 consensus_queue.cc:260] T 00000000000000000000000000000000 P d1a72b00159e4e9ca4a3dedc22af07e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9089a68b94bd4442bc7a5edbfb6ce048" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45909 } } peers { permanent_uuid: "9b75056b27df4d6c88b30ec8b99a0656" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38943 } } peers { permanent_uuid: "d1a72b00159e4e9ca4a3dedc22af07e4" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40187 } }
Sep 24 21:12:07 dist-test-slave-c7tk krb5kdc[22931](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284327, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260924 21:12:07.703796 23162 sys_catalog.cc:564] T 00000000000000000000000000000000 P d1a72b00159e4e9ca4a3dedc22af07e4 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 21:12:07.705062 23168 sys_catalog.cc:455] T 00000000000000000000000000000000 P d1a72b00159e4e9ca4a3dedc22af07e4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9089a68b94bd4442bc7a5edbfb6ce048" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45909 } } peers { permanent_uuid: "9b75056b27df4d6c88b30ec8b99a0656" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38943 } } peers { permanent_uuid: "d1a72b00159e4e9ca4a3dedc22af07e4" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40187 } } }
I20260924 21:12:07.705284 23168 sys_catalog.cc:458] T 00000000000000000000000000000000 P d1a72b00159e4e9ca4a3dedc22af07e4 [sys.catalog]: This master's current role is: FOLLOWER
W20260924 21:12:07.705475 23179 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d1a72b00159e4e9ca4a3dedc22af07e4: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 21:12:07.705526 23179 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 21:12:07.706662 23091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:12:07.699538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33808 (local address 127.0.98.61:38943)
0924 21:12:07.699576 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0924 21:12:07.699577 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 21:12:07.699630 (+ 53us) server_negotiation.cc:407] Connection header received
0924 21:12:07.699701 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:12:07.699701 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:12:07.699707 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:12:07.699743 (+ 36us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:12:07.700339 (+ 596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:12:07.700812 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:12:07.701650 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:12:07.701875 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:12:07.701932 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:12:07.704022 (+ 2090us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 21:12:07.704023 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 21:12:07.704023 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 21:12:07.704029 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 21:12:07.705536 (+ 1507us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:12:07.705904 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:12:07.705905 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:12:07.705906 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:12:07.705978 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:12:07.706212 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:12:07.706213 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:12:07.706213 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:12:07.706321 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 21:12:07.706456 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0924 21:12:07.706516 (+ 60us) server_negotiation.cc:299] Negotiation successful
0924 21:12:07.706580 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260924 21:12:07.707896 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 21:12:07.705672 23094 master_runner.cc:186] Error getting master registration for 127.0.98.62:45909: OK, instance_id { permanent_uuid: "9089a68b94bd4442bc7a5edbfb6ce048" instance_seqno: 1790284327401512 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260924 21:12:07.708240 23094 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260924 21:12:07.708959 22986 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "9089a68b94bd4442bc7a5edbfb6ce048"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35968
I20260924 21:12:07.741186 23012 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40187 } attempt: 2
Sep 24 21:12:07 dist-test-slave-c7tk krb5kdc[22931](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284327, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260924 21:12:07.747537 23014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:12:07.741460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40187 (local address 127.0.0.1:56526)
0924 21:12:07.741506 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0924 21:12:07.741516 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 21:12:07.741551 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 21:12:07.741729 (+ 178us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 21:12:07.741730 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 21:12:07.741925 (+ 195us) client_negotiation.cc:189] Negotiated authn=SASL
0924 21:12:07.741961 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:12:07.741962 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:12:07.742564 (+ 602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:12:07.742565 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:12:07.743006 (+ 441us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:12:07.743007 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:12:07.743298 (+ 291us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:12:07.743298 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:12:07.743347 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:12:07.743691 (+ 344us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 21:12:07.743696 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 21:12:07.744979 (+ 1283us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 21:12:07.746615 (+ 1636us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 21:12:07.746616 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 21:12:07.746616 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 21:12:07.746756 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 21:12:07.746929 (+ 173us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 21:12:07.746929 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 21:12:07.746930 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 21:12:07.747005 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 21:12:07.747262 (+ 257us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 21:12:07.747263 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 21:12:07.747301 (+ 38us) client_negotiation.cc:769] Sending connection context
0924 21:12:07.747373 (+ 72us) client_negotiation.cc:240] Negotiation successful
0924 21:12:07.747456 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260924 21:12:07.747545 23165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:12:07.741548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56526 (local address 127.0.98.60:40187)
0924 21:12:07.741600 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0924 21:12:07.741602 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 21:12:07.741610 (+ 8us) server_negotiation.cc:407] Connection header received
0924 21:12:07.741636 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:12:07.741637 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:12:07.741644 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:12:07.741721 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:12:07.742050 (+ 329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:12:07.742482 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:12:07.743080 (+ 598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:12:07.743235 (+ 155us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:12:07.743299 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:12:07.745057 (+ 1758us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 21:12:07.745058 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 21:12:07.745059 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 21:12:07.745064 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 21:12:07.746545 (+ 1481us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:12:07.746819 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:12:07.746820 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:12:07.746820 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:12:07.746866 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:12:07.747076 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:12:07.747076 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:12:07.747077 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:12:07.747193 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 21:12:07.747259 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0924 21:12:07.747404 (+ 145us) server_negotiation.cc:299] Negotiation successful
0924 21:12:07.747464 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260924 21:12:07.749145 23012 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9089a68b94bd4442bc7a5edbfb6ce048: Bootstrap starting.
I20260924 21:12:07.749827 23012 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9089a68b94bd4442bc7a5edbfb6ce048: Neither blocks nor log segments found. Creating new log.
I20260924 21:12:07.750110 23012 log.cc:824] T 00000000000000000000000000000000 P 9089a68b94bd4442bc7a5edbfb6ce048: Log is configured to *not* fsync() on all Append() calls
I20260924 21:12:07.750720 23012 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9089a68b94bd4442bc7a5edbfb6ce048: No bootstrap required, opened a new log
I20260924 21:12:07.751667 23012 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9089a68b94bd4442bc7a5edbfb6ce048 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9089a68b94bd4442bc7a5edbfb6ce048" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45909 } } peers { permanent_uuid: "9b75056b27df4d6c88b30ec8b99a0656" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38943 } } peers { permanent_uuid: "d1a72b00159e4e9ca4a3dedc22af07e4" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40187 } }
I20260924 21:12:07.751791 23012 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9089a68b94bd4442bc7a5edbfb6ce048 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9089a68b94bd4442bc7a5edbfb6ce048, State: Initialized, Role: FOLLOWER
I20260924 21:12:07.751875 23012 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9089a68b94bd4442bc7a5edbfb6ce048 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9089a68b94bd4442bc7a5edbfb6ce048" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45909 } } peers { permanent_uuid: "9b75056b27df4d6c88b30ec8b99a0656" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38943 } } peers { permanent_uuid: "d1a72b00159e4e9ca4a3dedc22af07e4" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40187 } }
I20260924 21:12:07.752161 23012 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9089a68b94bd4442bc7a5edbfb6ce048 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 21:12:07.752195 22986 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9089a68b94bd4442bc7a5edbfb6ce048 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260924 21:12:07.752272 22986 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9089a68b94bd4442bc7a5edbfb6ce048 [term 0 FOLLOWER]: Advancing to term 1
I20260924 21:12:07.753083 22986 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9089a68b94bd4442bc7a5edbfb6ce048 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9089a68b94bd4442bc7a5edbfb6ce048" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45909 } } peers { permanent_uuid: "9b75056b27df4d6c88b30ec8b99a0656" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38943 } } peers { permanent_uuid: "d1a72b00159e4e9ca4a3dedc22af07e4" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40187 } }
I20260924 21:12:07.753314 22986 leader_election.cc:288] T 00000000000000000000000000000000 P 9089a68b94bd4442bc7a5edbfb6ce048 [CANDIDATE]: Term 1 election: Requested vote from peers 9b75056b27df4d6c88b30ec8b99a0656 (127.0.98.61:38943), d1a72b00159e4e9ca4a3dedc22af07e4 (127.0.98.60:40187)
I20260924 21:12:07.752143 23182 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9089a68b94bd4442bc7a5edbfb6ce048 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9089a68b94bd4442bc7a5edbfb6ce048" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45909 } } peers { permanent_uuid: "9b75056b27df4d6c88b30ec8b99a0656" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38943 } } peers { permanent_uuid: "d1a72b00159e4e9ca4a3dedc22af07e4" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40187 } } }
I20260924 21:12:07.753724 23136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9089a68b94bd4442bc7a5edbfb6ce048" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "d1a72b00159e4e9ca4a3dedc22af07e4"
I20260924 21:12:07.753863 23136 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d1a72b00159e4e9ca4a3dedc22af07e4 [term 0 FOLLOWER]: Advancing to term 1
I20260924 21:12:07.753587 23182 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9089a68b94bd4442bc7a5edbfb6ce048 [sys.catalog]: This master's current role is: FOLLOWER
W20260924 21:12:07.754278 23193 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9089a68b94bd4442bc7a5edbfb6ce048: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 21:12:07.754393 23193 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 21:12:07.754604 23136 raft_consensus.cc:2454] T 00000000000000000000000000000000 P d1a72b00159e4e9ca4a3dedc22af07e4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9089a68b94bd4442bc7a5edbfb6ce048 in term 1.
I20260924 21:12:07.754791 22963 leader_election.cc:302] T 00000000000000000000000000000000 P 9089a68b94bd4442bc7a5edbfb6ce048 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9089a68b94bd4442bc7a5edbfb6ce048, d1a72b00159e4e9ca4a3dedc22af07e4; no voters:
I20260924 21:12:07.754933 23182 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9089a68b94bd4442bc7a5edbfb6ce048 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 21:12:07.755079 23182 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9089a68b94bd4442bc7a5edbfb6ce048 [term 1 LEADER]: Becoming Leader. State: Replica: 9089a68b94bd4442bc7a5edbfb6ce048, State: Running, Role: LEADER
I20260924 21:12:07.755182 23182 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9089a68b94bd4442bc7a5edbfb6ce048 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9089a68b94bd4442bc7a5edbfb6ce048" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45909 } } peers { permanent_uuid: "9b75056b27df4d6c88b30ec8b99a0656" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38943 } } peers { permanent_uuid: "d1a72b00159e4e9ca4a3dedc22af07e4" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40187 } }
I20260924 21:12:07.755548 23182 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9089a68b94bd4442bc7a5edbfb6ce048 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9089a68b94bd4442bc7a5edbfb6ce048. Latest consensus state: current_term: 1 leader_uuid: "9089a68b94bd4442bc7a5edbfb6ce048" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9089a68b94bd4442bc7a5edbfb6ce048" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45909 } } peers { permanent_uuid: "9b75056b27df4d6c88b30ec8b99a0656" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38943 } } peers { permanent_uuid: "d1a72b00159e4e9ca4a3dedc22af07e4" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40187 } } }
I20260924 21:12:07.755649 23182 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9089a68b94bd4442bc7a5edbfb6ce048 [sys.catalog]: This master's current role is: LEADER
I20260924 21:12:07.755034 22944 master_runner.cc:186] Error getting master registration for 127.0.98.61:38943: OK, instance_id { permanent_uuid: "9b75056b27df4d6c88b30ec8b99a0656" instance_seqno: 1790284327533352 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260924 21:12:07.757256 22944 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260924 21:12:07.760017 23201 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 21:12:07.763566 23165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:12:07.756292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56536 (local address 127.0.98.60:40187)
0924 21:12:07.756318 (+ 26us) server_negotiation.cc:206] Beginning negotiation
0924 21:12:07.756319 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 21:12:07.756556 (+ 237us) server_negotiation.cc:407] Connection header received
0924 21:12:07.756637 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:12:07.756638 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:12:07.756646 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:12:07.756670 (+ 24us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:12:07.757290 (+ 620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:12:07.757919 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:12:07.758851 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:12:07.759098 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:12:07.759130 (+ 32us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:12:07.760738 (+ 1608us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 21:12:07.760739 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 21:12:07.760739 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 21:12:07.760745 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 21:12:07.762377 (+ 1632us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:12:07.762850 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:12:07.762851 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:12:07.762851 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:12:07.762904 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:12:07.763082 (+ 178us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:12:07.763083 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:12:07.763083 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:12:07.763192 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 21:12:07.763251 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0924 21:12:07.763422 (+ 171us) server_negotiation.cc:299] Negotiation successful
0924 21:12:07.763490 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13}
I20260924 21:12:07.763780 23015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:12:07.757328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35982 (local address 127.0.98.62:45909)
0924 21:12:07.757357 (+ 29us) server_negotiation.cc:206] Beginning negotiation
0924 21:12:07.757358 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 21:12:07.757364 (+ 6us) server_negotiation.cc:407] Connection header received
0924 21:12:07.757379 (+ 15us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:12:07.757379 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:12:07.757397 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:12:07.757440 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:12:07.758025 (+ 585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:12:07.758611 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:12:07.759438 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:12:07.759686 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:12:07.759760 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:12:07.760954 (+ 1194us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 21:12:07.760955 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 21:12:07.760955 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 21:12:07.760961 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 21:12:07.762634 (+ 1673us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:12:07.763003 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:12:07.763004 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:12:07.763004 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:12:07.763045 (+ 41us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:12:07.763199 (+ 154us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:12:07.763200 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:12:07.763200 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:12:07.763300 (+ 100us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 21:12:07.763370 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0924 21:12:07.763635 (+ 265us) server_negotiation.cc:299] Negotiation successful
0924 21:12:07.763699 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260924 21:12:07.763751 23087 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40187 } attempt: 3
I20260924 21:12:07.762768 23201 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 21:12:07.765465 23136 raft_consensus.cc:1261] T 00000000000000000000000000000000 P d1a72b00159e4e9ca4a3dedc22af07e4 [term 1 FOLLOWER]: Refusing update from remote peer 9089a68b94bd4442bc7a5edbfb6ce048: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 21:12:07.765700 23195 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 9089a68b94bd4442bc7a5edbfb6ce048 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1a72b00159e4e9ca4a3dedc22af07e4" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 21:12:07.766460 23091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:12:07.755980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33816 (local address 127.0.98.61:38943)
0924 21:12:07.756016 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0924 21:12:07.756017 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 21:12:07.756219 (+ 202us) server_negotiation.cc:407] Connection header received
0924 21:12:07.756313 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:12:07.756313 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:12:07.756322 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:12:07.756358 (+ 36us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:12:07.757233 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:12:07.757866 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:12:07.759124 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:12:07.759681 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:12:07.759721 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:12:07.762968 (+ 3247us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 21:12:07.762970 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 21:12:07.762970 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 21:12:07.762975 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 21:12:07.764842 (+ 1867us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:12:07.765341 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:12:07.765342 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:12:07.765342 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:12:07.765402 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:12:07.765729 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:12:07.765730 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:12:07.765730 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:12:07.765867 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 21:12:07.765942 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0924 21:12:07.766314 (+ 372us) server_negotiation.cc:299] Negotiation successful
0924 21:12:07.766382 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260924 21:12:07.766825 23168 sys_catalog.cc:455] T 00000000000000000000000000000000 P d1a72b00159e4e9ca4a3dedc22af07e4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9089a68b94bd4442bc7a5edbfb6ce048. Latest consensus state: current_term: 1 leader_uuid: "9089a68b94bd4442bc7a5edbfb6ce048" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9089a68b94bd4442bc7a5edbfb6ce048" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45909 } } peers { permanent_uuid: "9b75056b27df4d6c88b30ec8b99a0656" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38943 } } peers { permanent_uuid: "d1a72b00159e4e9ca4a3dedc22af07e4" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40187 } } }
I20260924 21:12:07.766929 23168 sys_catalog.cc:458] T 00000000000000000000000000000000 P d1a72b00159e4e9ca4a3dedc22af07e4 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 21:12:07.768136 23195 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9089a68b94bd4442bc7a5edbfb6ce048 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9089a68b94bd4442bc7a5edbfb6ce048" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9089a68b94bd4442bc7a5edbfb6ce048" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45909 } } peers { permanent_uuid: "9b75056b27df4d6c88b30ec8b99a0656" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38943 } } peers { permanent_uuid: "d1a72b00159e4e9ca4a3dedc22af07e4" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40187 } } }
I20260924 21:12:07.768208 23195 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9089a68b94bd4442bc7a5edbfb6ce048 [sys.catalog]: This master's current role is: LEADER
I20260924 21:12:07.768255 23168 sys_catalog.cc:455] T 00000000000000000000000000000000 P d1a72b00159e4e9ca4a3dedc22af07e4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9089a68b94bd4442bc7a5edbfb6ce048" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9089a68b94bd4442bc7a5edbfb6ce048" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45909 } } peers { permanent_uuid: "9b75056b27df4d6c88b30ec8b99a0656" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38943 } } peers { permanent_uuid: "d1a72b00159e4e9ca4a3dedc22af07e4" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40187 } } }
I20260924 21:12:07.768335 23168 sys_catalog.cc:458] T 00000000000000000000000000000000 P d1a72b00159e4e9ca4a3dedc22af07e4 [sys.catalog]: This master's current role is: FOLLOWER
Sep 24 21:12:07 dist-test-slave-c7tk krb5kdc[22931](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790284327, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260924 21:12:07.768669 23205 mvcc.cc:205] Tried to move back new op lower bound from 7333004606525390848 to 7333004606485966848. Current Snapshot: MvccSnapshot[applied={T|T < 7333004606525390848}]
I20260924 21:12:07.769074 23201 catalog_manager.cc:1347] Generated new cluster ID: 48668e7f1cbe486eb01695660aba08fb
I20260924 21:12:07.769112 23201 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 21:12:07.771183 23089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:12:07.763944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40187 (local address 127.0.0.1:56552)
0924 21:12:07.764013 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0924 21:12:07.764022 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 21:12:07.764062 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 21:12:07.764267 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 21:12:07.764267 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 21:12:07.764427 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0924 21:12:07.764462 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:12:07.764463 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:12:07.765165 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:12:07.765166 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:12:07.766269 (+ 1103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 21:12:07.766272 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 21:12:07.767026 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 21:12:07.767027 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 21:12:07.767081 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:12:07.767610 (+ 529us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 21:12:07.767613 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0924 21:12:07.768955 (+ 1342us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 21:12:07.770417 (+ 1462us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 21:12:07.770417 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 21:12:07.770418 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 21:12:07.770576 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 21:12:07.770711 (+ 135us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 21:12:07.770711 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 21:12:07.770712 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 21:12:07.770749 (+ 37us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 21:12:07.770942 (+ 193us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 21:12:07.770943 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 21:12:07.770974 (+ 31us) client_negotiation.cc:769] Sending connection context
0924 21:12:07.771019 (+ 45us) client_negotiation.cc:240] Negotiation successful
0924 21:12:07.771099 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56}
I20260924 21:12:07.771461 23165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:12:07.764124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56552 (local address 127.0.98.60:40187)
0924 21:12:07.764163 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0924 21:12:07.764164 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 21:12:07.764169 (+ 5us) server_negotiation.cc:407] Connection header received
0924 21:12:07.764184 (+ 15us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:12:07.764185 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:12:07.764192 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:12:07.764230 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:12:07.764539 (+ 309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:12:07.765080 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:12:07.766434 (+ 1354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:12:07.766683 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:12:07.766740 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:12:07.769036 (+ 2296us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 21:12:07.769037 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 21:12:07.769037 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 21:12:07.769043 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 21:12:07.770346 (+ 1303us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:12:07.770629 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:12:07.770630 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:12:07.770630 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:12:07.770666 (+ 36us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 21:12:07.770791 (+ 125us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 21:12:07.770791 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 21:12:07.770791 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 21:12:07.770873 (+ 82us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 21:12:07.770924 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0924 21:12:07.771328 (+ 404us) server_negotiation.cc:299] Negotiation successful
0924 21:12:07.771387 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260924 21:12:07.772574 23087 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9b75056b27df4d6c88b30ec8b99a0656: Bootstrap starting.
I20260924 21:12:07.773186 23087 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9b75056b27df4d6c88b30ec8b99a0656: Neither blocks nor log segments found. Creating new log.
I20260924 21:12:07.773442 23087 log.cc:824] T 00000000000000000000000000000000 P 9b75056b27df4d6c88b30ec8b99a0656: Log is configured to *not* fsync() on all Append() calls
I20260924 21:12:07.773864 23206 mvcc.cc:205] Tried to move back new op lower bound from 7333004606525390848 to 7333004606485966848. Current Snapshot: MvccSnapshot[applied={T|T < 7333004606525390848}]
I20260924 21:12:07.774252 23087 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9b75056b27df4d6c88b30ec8b99a0656: No bootstrap required, opened a new log
I20260924 21:12:07.775761 23087 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9b75056b27df4d6c88b30ec8b99a0656 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9089a68b94bd4442bc7a5edbfb6ce048" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45909 } } peers { permanent_uuid: "9b75056b27df4d6c88b30ec8b99a0656" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38943 } } peers { permanent_uuid: "d1a72b00159e4e9ca4a3dedc22af07e4" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40187 } }
I20260924 21:12:07.775863 23087 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9b75056b27df4d6c88b30ec8b99a0656 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9b75056b27df4d6c88b30ec8b99a0656, State: Initialized, Role: FOLLOWER
I20260924 21:12:07.775926 23087 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9b75056b27df4d6c88b30ec8b99a0656 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9089a68b94bd4442bc7a5edbfb6ce048" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45909 } } peers { permanent_uuid: "9b75056b27df4d6c88b30ec8b99a0656" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38943 } } peers { permanent_uuid: "d1a72b00159e4e9ca4a3dedc22af07e4" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40187 } }
I20260924 21:12:07.776168 23087 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9b75056b27df4d6c88b30ec8b99a0656 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 21:12:07.776283 23061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9089a68b94bd4442bc7a5edbfb6ce048" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "9b75056b27df4d6c88b30ec8b99a0656"
I20260924 21:12:07.776598 23060 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9b75056b27df4d6c88b30ec8b99a0656 [term 0 FOLLOWER]: Advancing to term 1
I20260924 21:12:07.777339 23060 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 9b75056b27df4d6c88b30ec8b99a0656 [term 1 FOLLOWER]: Refusing update from remote peer 9089a68b94bd4442bc7a5edbfb6ce048: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 21:12:07.777626 23182 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 9089a68b94bd4442bc7a5edbfb6ce048 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9b75056b27df4d6c88b30ec8b99a0656" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38943 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 21:12:07.778167 23061 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 9b75056b27df4d6c88b30ec8b99a0656 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9089a68b94bd4442bc7a5edbfb6ce048 in term 1.
I20260924 21:12:07.778302 23211 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9b75056b27df4d6c88b30ec8b99a0656 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9089a68b94bd4442bc7a5edbfb6ce048" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45909 } } peers { permanent_uuid: "9b75056b27df4d6c88b30ec8b99a0656" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38943 } } peers { permanent_uuid: "d1a72b00159e4e9ca4a3dedc22af07e4" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40187 } } }
I20260924 21:12:07.778390 23211 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9b75056b27df4d6c88b30ec8b99a0656 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 21:12:07.779508 23201 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 21:12:07.779610 23211 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9b75056b27df4d6c88b30ec8b99a0656 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9089a68b94bd4442bc7a5edbfb6ce048. Latest consensus state: current_term: 1 leader_uuid: "9089a68b94bd4442bc7a5edbfb6ce048" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9089a68b94bd4442bc7a5edbfb6ce048" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45909 } } peers { permanent_uuid: "9b75056b27df4d6c88b30ec8b99a0656" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38943 } } peers { permanent_uuid: "d1a72b00159e4e9ca4a3dedc22af07e4" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40187 } } }
I20260924 21:12:07.779675 23211 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9b75056b27df4d6c88b30ec8b99a0656 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 21:12:07.779703 23211 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9b75056b27df4d6c88b30ec8b99a0656 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9089a68b94bd4442bc7a5edbfb6ce048" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9089a68b94bd4442bc7a5edbfb6ce048" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45909 } } peers { permanent_uuid: "9b75056b27df4d6c88b30ec8b99a0656" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38943 } } peers { permanent_uuid: "d1a72b00159e4e9ca4a3dedc22af07e4" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40187 } } }
I20260924 21:12:07.779744 23211 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9b75056b27df4d6c88b30ec8b99a0656 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 21:12:07.780097 23201 catalog_manager.cc:1504] Loading token signing keys...
I20260924 21:12:07.780263 23223 mvcc.cc:205] Tried to move back new op lower bound from 7333004606525390848 to 7333004606485966848. Current Snapshot: MvccSnapshot[applied={T|T < 7333004606525390848}]
I20260924 21:12:07.785043 23182 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9089a68b94bd4442bc7a5edbfb6ce048 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9089a68b94bd4442bc7a5edbfb6ce048" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9089a68b94bd4442bc7a5edbfb6ce048" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45909 } } peers { permanent_uuid: "9b75056b27df4d6c88b30ec8b99a0656" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38943 } } peers { permanent_uuid: "d1a72b00159e4e9ca4a3dedc22af07e4" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40187 } } }
I20260924 21:12:07.785115 23182 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9089a68b94bd4442bc7a5edbfb6ce048 [sys.catalog]: This master's current role is: LEADER
I20260924 21:12:07.785545 23225 catalog_manager.cc:1259] Loaded cluster ID: 48668e7f1cbe486eb01695660aba08fb
I20260924 21:12:07.785593 23225 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 9b75056b27df4d6c88b30ec8b99a0656: loading cluster ID for follower catalog manager: success
W20260924 21:12:07.785674 23225 catalog_manager.cc:1576] T 00000000000000000000000000000000 P 9b75056b27df4d6c88b30ec8b99a0656: acquiring CA information for follower catalog manager: Not found: root CA entry not found
W20260924 21:12:07.785712 23225 catalog_manager.cc:873] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry
I20260924 21:12:07.786603 23201 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9089a68b94bd4442bc7a5edbfb6ce048: Generated new TSK 0
I20260924 21:12:07.786755 23201 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260924 21:12:07.807652 23165 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 21:12:07.806430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56560 (local address 127.0.98.60:40187)
0924 21:12:07.806461 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0924 21:12:07.806462 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 21:12:07.807126 (+ 664us) server_negotiation.cc:407] Connection header received
0924 21:12:07.807206 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:12:07.807207 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:12:07.807215 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:12:07.807239 (+ 24us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:12:07.807605 (+ 366us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56560: BlockingRecv error: recv got EOF from 127.0.0.1:56560 (error 108)
Metrics: {"server-negotiator.queue_time_us":15}
W20260924 21:12:07.808507 23015 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 21:12:07.806706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35996 (local address 127.0.98.62:45909)
0924 21:12:07.806735 (+ 29us) server_negotiation.cc:206] Beginning negotiation
0924 21:12:07.806737 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 21:12:07.807818 (+ 1081us) server_negotiation.cc:407] Connection header received
0924 21:12:07.808037 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:12:07.808038 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:12:07.808046 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:12:07.808070 (+ 24us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:12:07.808460 (+ 390us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35996: BlockingRecv error: recv got EOF from 127.0.0.1:35996 (error 108)
Metrics: {"server-negotiator.queue_time_us":15}
W20260924 21:12:07.808876 23091 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 21:12:07.806910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33824 (local address 127.0.98.61:38943)
0924 21:12:07.806940 (+ 30us) server_negotiation.cc:206] Beginning negotiation
0924 21:12:07.806943 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 21:12:07.807919 (+ 976us) server_negotiation.cc:407] Connection header received
0924 21:12:07.808115 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:12:07.808116 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:12:07.808124 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:12:07.808147 (+ 23us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:12:07.808824 (+ 677us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33824: BlockingRecv error: recv got EOF from 127.0.0.1:33824 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260924 21:12:07.810810 23091 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 21:12:07.810182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33838 (local address 127.0.98.61:38943)
0924 21:12:07.810210 (+ 28us) server_negotiation.cc:206] Beginning negotiation
0924 21:12:07.810212 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 21:12:07.810384 (+ 172us) server_negotiation.cc:407] Connection header received
0924 21:12:07.810481 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:12:07.810482 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:12:07.810494 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:12:07.810517 (+ 23us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:12:07.810776 (+ 259us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33838: BlockingRecv error: recv got EOF from 127.0.0.1:33838 (error 108)
Metrics: {"server-negotiator.queue_time_us":15}
W20260924 21:12:07.813496 23091 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 21:12:07.812829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33852 (local address 127.0.98.61:38943)
0924 21:12:07.812857 (+ 28us) server_negotiation.cc:206] Beginning negotiation
0924 21:12:07.812859 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 21:12:07.813065 (+ 206us) server_negotiation.cc:407] Connection header received
0924 21:12:07.813150 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:12:07.813151 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:12:07.813159 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:12:07.813187 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:12:07.813462 (+ 275us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33852: BlockingRecv error: recv got EOF from 127.0.0.1:33852 (error 108)
Metrics: {"server-negotiator.queue_time_us":15}
W20260924 21:12:07.830350 23091 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 21:12:07.826036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33856 (local address 127.0.98.61:38943)
0924 21:12:07.826072 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0924 21:12:07.826074 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 21:12:07.829938 (+ 3864us) server_negotiation.cc:407] Connection header received
0924 21:12:07.830021 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:12:07.830022 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:12:07.830030 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:12:07.830054 (+ 24us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:12:07.830316 (+ 262us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33856: BlockingRecv error: recv got EOF from 127.0.0.1:33856 (error 108)
Metrics: {"server-negotiator.queue_time_us":14}
W20260924 21:12:07.835772 23091 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 21:12:07.835177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33868 (local address 127.0.98.61:38943)
0924 21:12:07.835203 (+ 26us) server_negotiation.cc:206] Beginning negotiation
0924 21:12:07.835204 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 21:12:07.835384 (+ 180us) server_negotiation.cc:407] Connection header received
0924 21:12:07.835472 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:12:07.835473 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:12:07.835480 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:12:07.835504 (+ 24us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:12:07.835739 (+ 235us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33868: BlockingRecv error: recv got EOF from 127.0.0.1:33868 (error 108)
Metrics: {"server-negotiator.queue_time_us":14}
W20260924 21:12:07.845268 23091 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 21:12:07.844642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33882 (local address 127.0.98.61:38943)
0924 21:12:07.844670 (+ 28us) server_negotiation.cc:206] Beginning negotiation
0924 21:12:07.844671 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 21:12:07.844868 (+ 197us) server_negotiation.cc:407] Connection header received
0924 21:12:07.844947 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:12:07.844948 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:12:07.844956 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:12:07.844980 (+ 24us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:12:07.845222 (+ 242us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33882: BlockingRecv error: recv got EOF from 127.0.0.1:33882 (error 108)
Metrics: {"server-negotiator.queue_time_us":14}
W20260924 21:12:07.863267 23091 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 21:12:07.862362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33886 (local address 127.0.98.61:38943)
0924 21:12:07.862421 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0924 21:12:07.862423 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 21:12:07.862737 (+ 314us) server_negotiation.cc:407] Connection header received
0924 21:12:07.862848 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:12:07.862849 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:12:07.862860 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:12:07.862890 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:12:07.863220 (+ 330us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33886: BlockingRecv error: recv got EOF from 127.0.0.1:33886 (error 108)
Metrics: {"server-negotiator.queue_time_us":31}
W20260924 21:12:07.897271 23091 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 21:12:07.896541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33902 (local address 127.0.98.61:38943)
0924 21:12:07.896585 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0924 21:12:07.896587 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 21:12:07.896708 (+ 121us) server_negotiation.cc:407] Connection header received
0924 21:12:07.896760 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:12:07.896761 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:12:07.896774 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:12:07.896850 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:12:07.897218 (+ 368us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33902: BlockingRecv error: recv got EOF from 127.0.0.1:33902 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260924 21:12:07.963061 23091 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 21:12:07.962373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33912 (local address 127.0.98.61:38943)
0924 21:12:07.962443 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0924 21:12:07.962445 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 21:12:07.962549 (+ 104us) server_negotiation.cc:407] Connection header received
0924 21:12:07.962631 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:12:07.962632 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:12:07.962642 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:12:07.962708 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:12:07.963005 (+ 297us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33912: BlockingRecv error: recv got EOF from 127.0.0.1:33912 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
W20260924 21:12:08.093216 23091 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 21:12:08.092223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33928 (local address 127.0.98.61:38943)
0924 21:12:08.092289 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0924 21:12:08.092291 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 21:12:08.092691 (+ 400us) server_negotiation.cc:407] Connection header received
0924 21:12:08.092767 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:12:08.092768 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:12:08.092779 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:12:08.092821 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:12:08.093130 (+ 309us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33928: BlockingRecv error: recv got EOF from 127.0.0.1:33928 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
W20260924 21:12:08.351106 23091 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 21:12:08.350361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33942 (local address 127.0.98.61:38943)
0924 21:12:08.350422 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0924 21:12:08.350424 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 21:12:08.350493 (+ 69us) server_negotiation.cc:407] Connection header received
0924 21:12:08.350551 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:12:08.350552 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:12:08.350560 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:12:08.350616 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0924 21:12:08.351042 (+ 426us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33942: BlockingRecv error: recv got EOF from 127.0.0.1:33942 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
I20260924 21:12:08.705981 23179 catalog_manager.cc:1259] Loaded cluster ID: 48668e7f1cbe486eb01695660aba08fb
I20260924 21:12:08.706089 23179 catalog_manager.cc:1552] T 00000000000000000000000000000000 P d1a72b00159e4e9ca4a3dedc22af07e4: loading cluster ID for follower catalog manager: success
I20260924 21:12:08.707161 23179 catalog_manager.cc:1574] T 00000000000000000000000000000000 P d1a72b00159e4e9ca4a3dedc22af07e4: acquiring CA information for follower catalog manager: success
I20260924 21:12:08.707444 23179 catalog_manager.cc:1602] T 00000000000000000000000000000000 P d1a72b00159e4e9ca4a3dedc22af07e4: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 21:12:08.786713 23225 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 9b75056b27df4d6c88b30ec8b99a0656: acquiring CA information for follower catalog manager: success
I20260924 21:12:08.787034 23225 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 9b75056b27df4d6c88b30ec8b99a0656: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 21:12:08.866639 23293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:12:08.864186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33952 (local address 127.0.98.61:38943)
0924 21:12:08.864312 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0924 21:12:08.864314 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 21:12:08.864363 (+ 49us) server_negotiation.cc:407] Connection header received
0924 21:12:08.864417 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:12:08.864418 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:12:08.864461 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:12:08.864526 (+ 65us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 21:12:08.864673 (+ 147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:12:08.865195 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:12:08.865991 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:12:08.866165 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:12:08.866213 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:12:08.866341 (+ 128us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 21:12:08.866435 (+ 94us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 21:12:08.866489 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0924 21:12:08.866516 (+ 27us) server_negotiation.cc:299] Negotiation successful
0924 21:12:08.866531 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107,"thread_start_us":71,"threads_started":1}
I20260924 21:12:08.870347 23299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:12:08.868242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56574 (local address 127.0.98.60:40187)
0924 21:12:08.868349 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0924 21:12:08.868350 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 21:12:08.868359 (+ 9us) server_negotiation.cc:407] Connection header received
0924 21:12:08.868401 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:12:08.868402 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:12:08.868448 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:12:08.868553 (+ 105us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 21:12:08.868632 (+ 79us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:12:08.869046 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:12:08.869649 (+ 603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:12:08.869852 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:12:08.869928 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:12:08.870065 (+ 137us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 21:12:08.870150 (+ 85us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 21:12:08.870222 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0924 21:12:08.870256 (+ 34us) server_negotiation.cc:299] Negotiation successful
0924 21:12:08.870272 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87,"thread_start_us":44,"threads_started":1}
I20260924 21:12:08.874230 23299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:12:08.871795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56578 (local address 127.0.98.60:40187)
0924 21:12:08.871856 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0924 21:12:08.871858 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 21:12:08.871964 (+ 106us) server_negotiation.cc:407] Connection header received
0924 21:12:08.872052 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:12:08.872053 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:12:08.872062 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:12:08.872091 (+ 29us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 21:12:08.872271 (+ 180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:12:08.872713 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:12:08.873580 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:12:08.873833 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:12:08.873884 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:12:08.873999 (+ 115us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 21:12:08.874027 (+ 28us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 21:12:08.874068 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0924 21:12:08.874147 (+ 79us) server_negotiation.cc:299] Negotiation successful
0924 21:12:08.874166 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260924 21:12:08.874542 23293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:12:08.871830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33968 (local address 127.0.98.61:38943)
0924 21:12:08.871872 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0924 21:12:08.871874 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 21:12:08.872000 (+ 126us) server_negotiation.cc:407] Connection header received
0924 21:12:08.872061 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:12:08.872061 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:12:08.872071 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:12:08.872128 (+ 57us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 21:12:08.872240 (+ 112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:12:08.872889 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:12:08.873706 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:12:08.873979 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:12:08.874044 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:12:08.874134 (+ 90us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 21:12:08.874157 (+ 23us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 21:12:08.874196 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0924 21:12:08.874470 (+ 274us) server_negotiation.cc:299] Negotiation successful
0924 21:12:08.874491 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260924 21:12:08.874595 23305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 21:12:08.871705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36002 (local address 127.0.98.62:45909)
0924 21:12:08.871809 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0924 21:12:08.871812 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 21:12:08.871963 (+ 151us) server_negotiation.cc:407] Connection header received
0924 21:12:08.872087 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 21:12:08.872088 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 21:12:08.872134 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 21:12:08.872186 (+ 52us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 21:12:08.872374 (+ 188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:12:08.873026 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:12:08.873680 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 21:12:08.873995 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 21:12:08.874045 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 21:12:08.874229 (+ 184us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 21:12:08.874307 (+ 78us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 21:12:08.874360 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0924 21:12:08.874496 (+ 136us) server_negotiation.cc:299] Negotiation successful
0924 21:12:08.874519 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80,"thread_start_us":50,"threads_started":1}
I20260924 21:12:08.876048 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAzNNjS/build/release/bin/kudu with pid 22944
I20260924 21:12:08.880857 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAzNNjS/build/release/bin/kudu with pid 23019
I20260924 21:12:08.885465 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAzNNjS/build/release/bin/kudu with pid 23094
2026-09-24T21:12:08Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (2618 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (5274 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (252092 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST